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APPENDIX A: Additional Background of the Evaluation 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program 


A-1 


This evaluation, which focuses on the capacity-building mission of the Comprehensive Centers, 
was designed to add to the findings of the previous national evaluation of the Comprehensive 
Centers Program. That evaluation, completed in 2011, used survey data, structured interviews, 
and rating scales to analyze the extent to which SEA capacity changed and to assess the quality, 
relevance, and usefulness of Center services across all state priority areas. The current 
evaluation analyzed the processes by which Centers may produce SEA capacity change. In 
addition, by focusing on two priority areas, the current evaluation aimed to provide more 
nuanced information on the implementation processes and how they may lead to changes in 
capacity-related outcomes. 


Comprehensive Centers and Their Context 


Title Il of the Educational Technical Assistance Act of 2002 (Section 203) authorized the 
Comprehensive Centers program. In 2012, the U.S. Department of Education awarded five-year 
cooperative agreement grants, later extended to seven years, to 15 Regional Centers and 7 
Content Centers under the program. Between 2012 and 2018, the U.S. Department of 
Education invested nearly $350 million in the 22 Comprehensive Technical Assistance (TA) 
Centers. These Centers were charged with delivering TA that builds the capacity of state 
education agencies (SEAs) to support local educational agencies (LEAs) in improving student 
outcomes. 


The 15 Regional Centers, assigned to regions as depicted on the map below, were required to 
build SEA capacity in the following federal priority areas, focusing on those that address each 
state’s particular needs: 


1. Implementing college and career-ready standards and aligned, high-quality assessments 
for all students 


2. Identifying, recruiting, developing, and retaining highly effective teachers and leaders 
3. Turning around the lowest-performing schools 


4. Ensuring the school readiness and success of preschool-age children and their successful 
transition to kindergarten 


5. Building rigorous instructional pathways that support the successful transition of all 
students from secondary education to college without the need for remediation, and 
careers 


6. Identifying and scaling up innovative approaches to teaching and learning that 
significantly improve student outcomes 


7. Using data-based decision making to improve instructional practices, policies, and 
student outcomes 


1 Turnbull, B. J., White, R. N., Sinclair, E., Riley, D. L., & Pistorino, C. (2011, August). National Evaluation of the 
Comprehensive Technical Assistance Centers: Final report (NCEE 2011-4031). Washington, DC: National Center for 
Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 
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In 2016, assistance with ESSA implementation was added as a priority area for the Centers’ work. 
Some Regional Centers were authorized to address additional priorities as appropriate for their 


regions, including Native American Education. 


The 15 Regional Centers served states and entities in their regions as follows: 


Appalachia Comprehensive Center: 


e §=Kentucky 

e Tennessee 

e §=6Virginia 

e West Virginia 


California Comprehensive Center: 
e = California 


Central Regional Comprehensive Center: 
e Colorado 
e Kansas 
e Missouri 


Florida and the Islands Comprehensive Center: 


e Florida 
e Puerto Rico 
e = Virgin Islands 


Great Lakes Comprehensive Center: 


e = Indiana 
e = Michigan 
e Ohio 
Mid-Atlantic Comprehensive Center: 
e Delaware 
e District of Columbia 
e Maryland 
e §6©New Jersey 
e ~=Pennsylvania 


Midwest Comprehensive Center: 


e Illinois 

e §=6lowa 

e Minnesota 
e Wisconsin 


North Central Comprehensive Center: 
e Nebraska 
e =North Dakota 
e South Dakota 
e Wyoming 


Northeast Comprehensive Center: 
e Connecticut 

Maine 

Massachusetts 

New Hampshire 

New York 

Rhode Island 

Vermont 


Northwest Comprehensive Center: 


e =©Alaska 

e Idaho 

e Montana 

e Oregon 

e Washington 


Pacific Comprehensive Center: 
e American Samoa 


e Commonwealth of the Northern Mariana 


Islands 
e Guam 
e =©Hawaii 
e =Palau 


South Central Comprehensive Center: 
e =©Arkansas 
e Louisiana 
e New Mexico 
e Oklahoma 


Southeast Regional Center: 
e Alabama 

Georgia 

Mississippi 

North Carolina 

South Carolina 


Texas Comprehensive Center: 
e Texas 


West Comprehensive Center: 


e §=6Arizona 
e Nevada 
e Utah 
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The seven Content Centers were required to deliver TA nationally in one of seven priority areas 
(the first six above plus one focusing generally on “building state capacity”), using their content 
expertise to build the capacity of SEAs and Regional Centers in that priority area. The seven 
Content Centers included: 


* Center on Building State Capacity and Productivity 

* Center on College and Career Readiness and Success 

* Center on Enhancing Early Learning Outcomes 

¢ Center on Innovations in Learning 

* Center on Standards and Assessments Implementation 
¢ Center on School Turnaround 

¢ Center on Great Teachers and Leaders 


SEAs’ Changing Needs for Technical Assistance 


Over the past few decades, SEAs have been asked to expand their traditional role of monitoring 
district and school compliance to include developing standards and accountability systems and 
improving school performance.’ Beginning in the 1990s, federal mandates have required the 
SEAs to take on expanded roles and responsibilities in implementing educational reforms.? 
Accumulating demands and expectations have driven the SEAs’ needs for TA from the 
Comprehensive Centers and other sources. Below we briefly describe major federal laws and 
policies that informed SEAs’ needs for TA during the evaluation period. 


The Structures of SEAs 

SEAs vary in their structure and approach to their work. Each agency is led by a chief, who may be 
called a superintendent, secretary, director, or commissioner of education or public instruction. In 
some states, governors appoint the chief, while in others the chief state school officer is elected. SEAs 
are operated under the authority granted by a state’s constitution, state laws and regulations, and 
federal law. In some states, a state board of education also exercises oversight and shares key 
responsibilities with the SEA. SEAs’ needs for TA could vary greatly due, in part, to their differing levels 


of influence on state education policy and differing levels of authority over districts.* For example, 
some SEAs administer the teacher evaluation systems in their states, while others have no role or only 
provide guidance or recommendations. 


*Gottfried, M. A., Stecher, B. M., Hoover, M., & Cross, A. B. (2011). Federal and state roles and capacity for improving 
schools. Retrieved from http://www.rand.org/content/dam/rand/pubs/technical_reports/2011/RAND TR989.pdf. Also 


see Aspen Institute. (2015). Roles and responsibilities of the state education agency. Retrieved from 
https://www.aspeninstitute.org/publications/roles-responsibilities-state-education-agency 


2 Murphy, P., & Hill, P. (2011). The changing role of states in education: The move from compliance to performance 
management (2011 PIE Network Summit Policy Briefs). Retrieved from 
http://www.crpe.org/sites/default/files/PIE_ Changing Role Oct11.pdf 

3 Ibid. 
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In the Improving America’s Schools Act of 1994, the reauthorization of the Elementary and 
Secondary Education Act (ESEA), Congress linked new requirements for state action to states’ 
eligibility for funding.* When ESEA was reauthorized in 2001, the President and Congress 
further strengthened its requirements through the No Child Left Behind Act (NCLB).° NCLB 
required states to adopt grade-level standards, administer state assessments in reading and 
math, and develop accountability systems. States faced significant implementation challenges 
with NCLB.® As subsequent federal programs as well as modifications to NCLB requirements 
unfolded, SEAs’ roles and responsibilities evolved further, sometimes with implications for 
their work in the two focal priority areas of this evaluation. 


In 2009, the federal government invested $4.35 billion in the Race to the Top (RTT) initiative, 
created with funds from the American Recovery and Reinvestment Act. RTT established a 
competitive grant program for states committed to implementing a comprehensive set of 
education reforms. Selection criteria for awarding funds included proposals for improving teacher 
and principal effectiveness.’ Through 2011, 19 states were awarded RTT grants for up to four 
years. In addition, 20 states received RTT Early Learning Challenge Grants to improve and increase 
access to early learning programs and services. RTT grants required complex planning and 
implementation, including in the two focal areas of this evaluation (teacher and leader 
effectiveness; early learning), which may have increased the need for TA among many SEAs. 


Between 2011 and 2014, 43 states received ESEA flexibility waivers from the U.S. Department 
of Education, waiving certain NCLB requirements and penalties.® In exchange for the waivers, 
states submitted plans for adopting and implementing reforms such as college- and career- 
ready standards and rigorous teacher and leader evaluation systems that include student 
achievement data. Administration of complex reforms required under these waivers 
necessitated new types of expertise among some SEAs receiving these waivers. 


In July 2014, the Department of Education notified SEAs of the requirement to submit new 
state educator equity plans in accordance with the requirements of Title | of the Elementary 
and Secondary Education Act. The plans had to describe how the states will ensure that 
disadvantaged students have the same level of access to highly qualified teachers as other 
students. To create the plans, SEAs were required to consult with key stakeholders (LEAs, 
teachers, principals, administrators, parents, and others), use data to define equity gaps, 
explain the root causes of the equity gaps, and describe how they intended to address the 
equity gaps and report on their progress.? These federal requirements further fueled SEAs 
needs for Centers’ TA, as evidenced by the fact that equity plan development and 


, 


* https://www2.ed.gov/offices/OESE/archives/legislation/ESEA/brochure/iasa-bro.html 


> No Child Left Behind Act of 2001, P.L. 107-110. https://www.congress.gov/bill/107th-congress/house-bill/1 
5 Goertz, M. E. (2005). Implementing the No Child Left Behind Act: Challenges for the states. Peabody Journal of 


Education, 80(2), 73-79. 

7 https://www2.ed.gov/programs/racetothetop/index.html 

8 http://www.edweek.org/ew/section/infographics/nclbwaivers.html 

° https://www2.ed.gov/programs/titleiparta/equitable/titleiiequityanalysis1031.pdf 
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implementation was the most common topic of SEAs’ work with Centers in the teacher/leader 
effectiveness priority area during the evaluation period (see Exhibit D-8). 


In December 2015, the Every Student Succeeds Act (ESSA) was signed into law, replacing 
NCLB.*° Compared to NCLB, ESSA gives SEAs more flexibility to tailor their own education 
policies. Under ESSA, states were required to develop and submit consolidated state plans in 
spring or fall 2017. States that previously had NCLB waivers no longer have to include student 
outcomes in educator evaluation systems. However, states have had to make a number of new 
decisions about standards, assessments, accountability indicators, plans for low-performing 
schools and parent involvement. 


During the evaluation period SEAs also faced financial and human resources shortages and 
cutbacks. At a time when ESSA increased SEAs’ responsibilities, SEAs in about half of the states 
were still coping with budget cuts and staff reductions dating back to the 2007-2009 economic 
recession." Even in states that have since increased education funding levels, more funding has 
been applied at local levels than at the state level.” The result was that SEAs continued to lay 
off or furlough staff or to leave vacant positions unfilled.+? In addition, the winding down of RTT 
grants during the latter part of the evaluation period may have contributed to shifting priorities, 
SEA staffing reductions in key priority areas, and further strain on resources in some states 
(Findings related to SEA staff shortages and turnover are explored in the report). 


All of these developments since the 1990s that created additional responsibilities for SEAs likely 
helped to create a greater need for TA more generally. The latest federal policies and programs 
over the past decade, in particular, probably helped shape SEAs’ specific needs for TA during 
the period of evaluation. 


Technical Assistance Resources Available to SEAs 

The National Comprehensive Technical Assistance Center program is one of many federal and 
non-federal programs that provide TA to SEAs. In contrast to most TA providers, the 
Comprehensive Centers address a wider range of policy areas and SEA needs. Below are 
descriptions of several major programs that provided TA to SEAs during the evaluation period 
and that sometimes collaborated with the Centers: 


* The Regional Educational Laboratories (RELs) are contracts funded by the U.S. 
Department of Education to conduct applied research projects and partner with school 
districts, SEAs, and others to build their capacity to use data and research-based 
evidence. The ten REL regions overlap with but do not perfectly correspond to the 
fifteen Comprehensive Center regions. Both RELs and Comprehensive Centers aim to 
build SEA capacity to improve student achievement, with the RELs focusing specifically 


10 http://www.ed.gov/essa?src=rn 

11 Burnette, D. (2016, March 11). ESSA poses capacity challenges for state education agencies. Education Week. Retrieved from 
https://www.edweek.org/ew/articles/2016/01/20/essa-poses-capacity-challenges-for-state-education.html 

2 Ibid. 

13 Kober, N., & Rentner, D. S. (2012). State education agency funding and staffing in the education reform era. Washington, DC: 
Center on Education Policy, George Washington University. 
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on research and analytic capacity and the Centers focusing on designing and 
implementing education reforms. 


° are grants 

funded by the U.S. Department of Education to 
Typical 

activities include TA in the selection of appropriate education programs for students 
with limited English proficiency or training designed to develop educators’ skills in 
specific areas. For example, the centers disseminate information on successful 
education practices and the legal requirements related to nondiscrimination in 
education programs. 


¢ The Early Learning Challenge Collaborative is a privately funded initiative that has 
supported states applying for and receiving Race To the Top Early Learning Challenge 
Grants, helping them through tailored TA, topical consortia, webinars, and learning 
community activities. 


Literature on Capacity Building 


In the evaluation’s initial phase of literature review, document review, and informal meetings 
with Centers,?° the research team examined existing concepts of capacity building in order to 
inform the development of interview protocols and coding. These concepts were the subject of 
continued examination and further refinement throughout the evaluation. 


The research team reviewed literature that described or defined the process and goals of 
capacity building in public agencies or systems, particularly in education. Some of these 
documents were suggested to the research team by Center staff during the informal meetings 
with the Centers. The team also conducted a database search of reports and articles, searching 
Google Scholar, JSTOR, and EBSCO for publications from 2003 to 2013 that focused on capacity 
building in education and in public reform. 


The following concepts emerged from the literature review and informed the study design: 


*  Century’s four types of “capacity within systemic educational reform.”'’ Century defines 
capacity as an entity’s “ability to achieve the goals of a reform” and articulated four 
dimensions of capacity necessary for reform: human, organizational, structural, and 
material. Human capacity requires people with “intellectual proficiency and will,” 
including knowledge, expertise, understanding, and persistence for carrying out reform. 


4 http://www2.ed.gov/programs/equitycenters/index.html 


15 http://www. buildinitiative.org/AboutUs/MissionHistory.aspx 

16 Evaluators met with Center staff early in the evaluation period in order to collect documents and request input 
on the evaluation design. 

1” Century, J. R. (1999, April). Determining capacity within systemic educational reform. Paper presented at the 
annual meeting of the American Educational Research Association, Montreal, Quebec, Canada. Retrieved from 
http://files.eric.ed.gov/fulltext/ED434162. pdf 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program A-7 


Organizational capacity involves interactions and communication among individuals in 
the system, forming an organizational culture. Structural capacity includes the elements 
of the system that exist independently of the individuals in the system, such as policies, 
practices, and processes, which may or may not be sustainable. Material capacity, which 
is less amenable to TA than the other dimensions, is made up of the financial and 
material resources and support available to the reform. 


* Harsh’s notions of “stages” of capacity building, culminating in sustainability.'® Harsh, a 
former director of the Appalachia Regional Comprehensive Center, describes capacity 
building that gradually moves organizations through the stages of exploration, emerging 
implementation, full implementation, and sustainability. In the final stage, the 
constituent organization is able to make “pervasive and consistent” use of the newly 
gained skills and practices and to self-analyze and modify practices when needed. 


*  Coburn’s requirements for bringing educational reforms to scale. 9 Coburn emphasizes 
four requirements for going to scale: “deep change,” or changes in beliefs, norms, and 
principles; “sustained” change, or change that continues over time; “spread” of change; 
and “ownership” of the reform by all the participants. 


¢ Studies of SEA capacity in specific policy areas have found widely varying state needs for 
TA. Massell et al.,2° focusing on how SEAs use research in school improvement policy, 
and McGuinn,”! looking at SEAs’ implementation of new teacher evaluation systems, 
found that SEAs had widely differing numbers and types of staff, organizational 
structures and rules, and relationships with external organizations, all of which affected 
their use of TA. 


This literature, along with Centers’ documents, was used to identify possible themes in the 
Centers’ approach to capacity building and to develop interview questions and constructs for 
coding, particularly for the design-focused interviews that took place in 2015. 


18 Harsh, S. (2010). Gaining perspective on a complex task: A multidimensional approach to capacity building. In S. 
Harsh, K. Bradley, K. Good, & J. Ross (eds.), Capacity building technical assistance: Change agent analyses (pp.1- 
19). Charlestown, WV: Edvantia, Appalachia Regional Comprehensive Center. 

19 Coburn, C. E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational 
Researcher, 32(6), 3-12. 

20 Massell, D., Goertz, M. E., & Barnes, C. (n.d.). State education agencies’ acquisition and use of research 
knowledge in schoo! improvement strategies. Retrieved from 
http://www.cpre.org/sites/default/files/journal/1421 pjemassellgoertzbarnesfinalversion625121.pdf 

1 McGuinn, P. (2012, November). The state of teacher evaluation reform: State education agency capacity and the 
implementation of new teacher evaluation systems. Retrieved from 
http://www.cpre.org/sites/default/files/researchreport/1495_mcguinnthestateofevaluation-final.pdf 
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APPENDIX B: Evaluation Methods 
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The evaluation was designed to combine quantitative data from surveys and activity reports, 
which collectively provided information on all projects in two focal priority areas, 
teacher/leader effectiveness and early learning, with extensive qualitative interview data. 
Interviews allowed close examination of Centers’ design process overall as well as the capacity- 
building process in a subset of projects in the two priority areas. 


A focus on two federal priority areas in the implementation and outcomes components of the 
evaluation enabled the evaluation to limit burden on the Centers while examining Centers’ TA 
in these areas in depth. The two priority areas were selected with input from the evaluation’s 
Technical Working Group and the U.S. Department of Education. The teacher/leader 
effectiveness area was chosen because it was a particularly high priority for the U.S. 
Department of Education at the start of the evaluation and because all Centers had multiple 
projects in this area. The early learning area was chosen because it was a newer or emerging 
priority area for SEAs, required different TA approaches than the teacher/leader effectiveness 
area, and potentially offered a contrast to teacher/leader effectiveness in implementation and 
outcomes. 


In addition to participating in survey data collection and activity data collection each year, all 
Centers participated in interviews addressing questions about the general design of their TA in 
2015, and provided documents including theories of action overviews and annual work plans. 
All Centers also participated in interviews about implementation (in both 2015 and 2016) and 
outcomes (in 2017) for selected projects in the two focal priority areas. Center staff selected 
projects for discussion in interviews, and these were not expected to be representative of the 
universe of projects in the two priority areas. 


Similar data from TA recipients were collected. For each of the three survey years, Centers 
identified the key TA recipients for all projects listed in the activity reports in the two focal 
priority areas and confirmed by the Centers as active for each survey period. These TA 
recipients were then surveyed on implementation and outcomes. Analogously, Center staff 
identified one or two key TA recipients for each of the projects discussed in Center interviews. 
These TA recipients were then individually interviewed on those projects in each of the three 
years of data collection. 


Survey data and activity data were analyzed through calculation of frequencies or average 
rating scores; results are presented in tables and charts in this report. Interviews were 
systematically coded and analyzed using qualitative analysis software. This process allowed 
examination and illustration of quantitative findings, as well as analysis of new themes that 
emerged in interviews. Exhibit B-1 lists the data sources used to answer each of the 
evaluation’s research questions. The remainder of this appendix provides additional details on 
each of these data sources. 
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Exhibit B-1. Research Questions and Data Sources 


Research Questions Data Sources 


Design 

How did the Centers define capacity building in their theories of action? Center staff interviews 
Center documents 

How did the Centers assess the needs of their constituents and develop work Center staff interviews 

plans to address those needs? Center documents 

Implementation 

What strategies did Centers employ to achieve their outcomes? Center staff interviews 


TA recipient interviews 
Center documents 
Center TA activity reports 
To what extent and how did Centers collaborate with each other? Center staff survey 
Center staff interviews 
Center TA activity reports 
Center documents 

What challenges did Centers face and how did they respond? Center staff survey 

TA recipient survey 
Center staff interviews 
TA recipient interviews 


Outcomes 


Did Centers achieve their expected capacity-building outcomes, and how did they | Center staff survey 
know? TA recipient survey 
Center staff interviews 
TA recipient interviews 
Center documents 
What strategies were perceived to be most effective and why? Center staff survey 
TA recipient survey 
Center staff interviews 
TA recipient interviews 


Center Staff and TA Recipient Surveys 


Survey instruments are included in Appendix C. 


Web-based surveys of Center staff and TA recipients were administered in spring 2015, 2016, and 
2017. The surveys were administered to Center staff and key TA recipients who were involved in 
projects in the two priority areas over the previous six months. Each year, evaluators worked with 
Center staff to develop updated lists of relevant projects and people who could report on those 
projects. All Center staff who worked on projects in the two priority areas were included in the 
survey sample. To form the TA recipient survey sample, Centers were asked to identify “key” TA 
recipients: those who were knowledgeable about each project and able to report on the capacity- 
building outcomes of the Centers’ TA. The guidance letter for Centers on preparing for the 
surveys is provided below. 


The surveys, consisting of both selected-response and open-ended items, included questions 
about implementation and outcomes. On the Center staff survey, Center staff members were 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program B-3 


asked to select the projects on which they had worked and then complete the same set of 
seven questions for each project they selected. For example, respondents were asked to rate 
the overall success of each project in building the capacity of its constituents by selecting a 
response on a four-point scale ranging from “not at all” to “to a great extent.” Respondents had 
the option of writing in explanations for their ratings. All data from the Center staff survey are 
at the project level. 


In contrast to the Center staff survey, TA recipients completed a single set of questions about 
all projects in which they had participated. To minimize burden, and because TA recipients 
might not explicitly distinguish and attribute the support they received from Centers to specific 
Center projects, TA recipients were asked to provide feedback on Center TA generally, after 
identifying the projects on which they had received Center TA. Most respondents had received 
TA on more than one project, and many received TA on both teacher/leader and early learning 
projects. The questions for TA recipients that provided data for this report included requests for 
ratings of Centers’ contributions to a number of specific improvements associated with 
individual, organizational, and policy development or implementation capacity. TA recipients 
were also asked to write in their responses about the most helpful services they received from 
Centers. (See detailed definitions of codes applied to write-in survey questions.) All data from 
the TA recipient survey are at the respondent level. 


Selected-response survey items were analyzed by tabulating each response option. Responses 
to open-ended survey questions were coded by a team of four coders. Responses were coded 
into existing categories where appropriate; for example, most responses to the request for 
“other” challenges on both surveys could be recoded into one of the categories of challenges 
already listed on the survey. To code data that did not fit into existing categories, coders used a 
multistep process. Coders first read through responses and created categories that captured 
recurring themes in the data. These categories were turned into codes through an iterative 
process, by checking and rechecking how common the responses were. The goals were to 
create codes broad enough to represent respondents’ major concerns and to capture 
constructs relevant to the research questions. If fewer than 10 responses ultimately fell into a 
code or category, coders re-examined the need for the code and considered consolidation with 
other codes. On the Center staff survey, four new categories of challenges were created based 
on write-in descriptions of “other,” and four categories of project collaborators were created 
and analyzed. No new categories of challenges were added to the list included on the TA 
recipient survey. 


The Center staff survey achieved a response rate of 79% (190 of 242 eligible staff) in 2015, 83% 
(194 of 235 eligible staff) in 2016, and 86% (166 of 193 eligible staff) in 2017. The TA recipient 


survey achieved a response rate of 67% (246 of 369 eligible TA recipients) in 2015, 68% (232 of 
343 eligible TA recipients) in 2016; and 68% (235 of 345 eligible TA recipients) in 2017.” 


22 Sample Ns reported in exhibits may differ from the total respondent Ns due to item-level non-response. 
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While Center staff survey data were aggregated to the project level across years, TA recipient 
survey data (which did not provide data at the project level) were analyzed at the respondent 
level, separately for each of the three survey years. TA recipient respondents selected from the 
project list, on average, three projects each per year. Their selections broke down by priority 
area as follows: projects in teacher/leader effectiveness only, 66% of TA recipients in 2015, 68% 
in 2016, and 65% in 2017; projects in early learning only, 22% of TA recipients in 2015, 21% in 
2016, and 22% in 2017. The remaining respondents (12% in 2015, 11% in 2016, and 13% in 
2017) chose projects in both priority areas. 


The surveys gathered respondents’ views on most projects in the two priority areas each year. 
Exhibit B-2 presents details on the percentages, out of all projects in the two priority areas 
implemented by the Centers each year, that were covered by the survey respondents, and also 
presents the mix of projects in the survey sample by Center type and priority area. Exhibit B-3 
presents the distribution of projects that were included in the survey for one, two, or three 
years. Projects varied in their timing and their duration. In interpreting the survey results, it is 
helpful to keep in mind that the data include projects with a mix of time periods. About 15% of 
all projects in the surveys appeared in the surveys for all three years of survey data collection. 


Exhibit B-2. Project Sample in Surveys 


Projects Covered in Surveys 


Total Project Projects 

Trey =a Mol ML) (Lad (oy MMy-)(-ax-tNe)'aa| Regional | Content |Mi-r\eil-\9 Mle <liNy7 

SUA A OC CLM eit] Centers | Centers |@i--\-(jamek-laniss 
Center staff survey, 2015 126 98% 123 82 41 95 28 
Center staff survey, 2016 166 93% 154 94 60 114 40 
Center staff survey, 2017 155 96% 149 98 51 114 35 
TA recipient survey, 2015 124 96% 119 79 40 92 27 
TA recipient survey, 2016 166 83% 138 82 56 107 31 
TA recipient survey, 2017 155 92% 143 93 50 108 35 


Source: Center staff and TA recipient surveys, 2015, 2016, and 2017. 

*Project selection rate is the percentage of projects on the list selected by at least one respondent. On the Center 
staff survey (individual response rate about 80% per year), respondents addressed a set of questions about each 
project they selected. On the TA recipient survey (response rate about 68% per year), respondents answered one set 
of questions about all projects they selected. TA recipients selected an average of three projects in each year. In 
2015, Center staff could not identify TA recipients for two projects; these projects focused on development of 
resources that were broadly disseminated online. 
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Exhibit B-3. Projects in Survey Data Across Three Years 


Number of Years in Which Regional Content Teacher/ Early 
Project Was Included in Survey Total* Centers Centers Leader Learning 
Data 

Center Staff Survey (N = 279) (N = 171) (N = 213) (N = 66) 
1 year 62% 57% 70% 63% 61% 
2 years 23% 25% 19% 23% 23% 
3 years 15% 18% 11% 15% 17% 
TA Recipient Survey (wy 4:3:)) (N = 217) 

1 year 62% 56% 70% 63% 58% 
2 years 23% 26% 19% 22% 27% 
3 years 15% 18% 11% 15% 15% 


Source: Center staff survey and TA recipient survey, 2015, 2016, and 2017. 
*Projects in the activity data (N = 333) have a similar distribution overall: 57% in the data for 1 year, 28% for 2 


years, and 15% for 3 years. 


Guidance Provided to Centers on Preparing for Surveys 


In preparation for the Center staff and TA recipient surveys each year, Center directors were 
provided an initial project list based on their annual work plans and were asked to confirm or 
correct the list, and to identify Center staff and TA recipients to be surveyed. For example, 
guidance to the Center directors in spring of 2015 included the following: 


The attached spreadsheet has three tabs: 1) Projects, 2) Center Staff, and 3) TA Recipients. 

The Projects tab contains a list of projects related to the selected priority areas that we 
identified from your Year 3 work plan. Please either confirm this list as accurate, or update the 
list by correcting project names or adding/deleting projects. The project names should be those 
that your staff and TA recipients would recognize. 


The Center Staff tab is formatted for collecting contact information for the online Center staff 
survey, but does not yet have any data. Please add contact information for all Center staff, sub- 


contractors and consultants who were involved in the work on any of the projects in the 


updated/confirmed list. You do not need to include administrative staff working on these projects. 
In some cases, staff members may work on more than one project in the list. Please include staff 
members who work on any of the projects on the list, regardless of how many they work on. 
During the survey, respondents will be able to select the projects they know about from the 


project list. 


The TA Recipients tab is formatted for collecting contact information for the online TA recipient 
survey, but does not yet have any data. Please add contact information for all individuals who 
were key recipients of, or participants in, technical assistance activities related to the projects in 


the two selected priority areas. These should be people who are the direct or primary 


participants in TA projects from your Center, and can knowledgeably comment on the TA and 
how it is being used. The TA recipients identified for our survey should be those who are 
expected to use the TA to build their own or others’ capacity to support local education agencies 
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(LEAs) and schools. Please use the guidance in the box below to determine how to select TA 
recipients for our survey sample. Some projects may not have key recipients in the way we are 


defining them. This is perfectly acceptable. 


TA Recipients on the list should include... But not... 


Staff of SEAs, Regional Centers (if you direct a Content 
Center), or other constituent organizations who could 
knowledgeably comment on the TA and related 
outcomes. 


People whose involvement is limited to implementing 
the programs or policies developed by SEAs and 
Comprehensive Center partners (but who do not have 
direct knowledge of the Centers). 


Constituents who have worked directly with your 
Center in planning, implementing, or overseeing the 
selected capacity-building projects and associated 
activities. 


LEA- and school-level staff who participate in needs- 
sensing events with no specific capacity building 
follow-on activity or outcomes expected. 


Constituents who are expected to build capacity in 
LEAs and schools after receiving TA from your Center. 


LEA- and school-level staff who participate in capacity 
building activities led by SEA staff, but not by your 
Center. 


Constituents who directly and actively participated in 
Center-led events or sponsored activities for the 
selected projects. Active participation includes 
engaging in work sessions and any follow-up activities. 


People whose participation was limited to listening in 
to webinars or attending occasional training events. 


Constituents who received Center consultations or 
responsive technical assistance over a period of 
months, working towards a significant product, policy 
change, or initiative. 


People whose only involvement with the Center was a 
one-time request for information. 


The number of TA recipients identified may vary for each project depending on its size and 
intended reach. We understand that in some cases, TA recipients may work with you on more 
than one project in the list. Please include key TA recipients who work with you on any of the 
projects on the list, regardless of how many they work on. During the survey, respondents will 
be able to select the projects they know about from the project list. 


Please let us know if you have questions. We are happy to set up a call with you to review these 


directions and assist you. 


Center Staff and TA Recipient Interviews 


Interview protocols are included in Appendix C. 


Center Interviews 


Evaluators went on annual site visits to conduct group interviews with Center staff in 2015, 
2016, and 2017. Each site visit team consisted of the site liaison (a research expert) and a TA 
expert. As the site visits were planned, each Center director was given the flexibility to 
determine which of his or her colleagues (up to five individuals) would be present for each 


interview. 
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The focus of interviews differed each year. Design-focused interviews were administered during 
site visits in 2015. Researchers asked about Center definitions of capacity building, theories of 
action, and approaches to needs assessment and planning. Respondents included some 
combination of the following Center staff members: director, deputy directors, evaluators, 
project leads, content leads, and TA providers. All 22 Centers participated in the 2015 Design- 
focused interviews. 


Implementation interviews were administered during site visits in 2015 and 2016. One or two 
group implementation interviews were conducted at each Center, depending on whether the 
Center was currently implementing projects in one or both of the focal priority areas in that 
given year. In preparation for the visits, the Center directors were asked to select one or two 
illustrative projects to discuss in each interview; they were also asked to invite up to five 
relevant staff to participate in each interview. Interview participants included TA managers and 
content specialists who worked in each priority area. Implementation interview questions 
focused on the goals, TA strategies, activities, challenges, progress, and early outcomes (when 
relevant) of the selected projects. The protocol also included several questions about initiatives 
and trends in the priority area. In both years, group interviews in the area of great 
teachers/leaders were conducted in all 22 Centers. Some Centers did not implement early 
learning projects in these years, so interviews were conducted in the 15 Centers with early 
learning projects in 2015 and in the 14 Centers with early learning projects in 2016. 


Outcomes interviews were administered during the 2017 site visits. One or two group outcomes 
interviews were conducted at each Center, depending on whether the Center implemented 
projects in one or both of the focal priority areas in 2016. In preparation for the visits, the Center 
directors were asked to identify one project per priority area that had been discussed in an 
implementation interview in 2016 (or in both 2015 and 2016). They were also asked to invite up 
to five relevant staff, including TA leaders for the project and Center evaluators, to participate. 
Outcomes interview questions focused on the outcomes, outcomes measures, challenges to 
achieving outcomes, and effective strategies for producing outcomes of the selected projects. 
The protocol also included several questions about outcomes, challenges, and strategies in the 
priority area overall. Group interviews on teacher/leader effectiveness projects were conducted 
in all 22 Centers. Group interviews were conducted in the 14 Centers that had discussed early 
learning projects in 2016. 


TA Recipient Interviews 

Design interviews, conducted in 2015, included Center staff only. Implementation interviews, 
conducted in 2015 and 2016, included both Center staff and TA recipients. Outcomes 
interviews, conducted in 2017, also included both Center staff and TA recipients. After the 
Center site visits, interviewers conducted telephone interviews with 1-2 TA recipients for each 
of the projects discussed during the Center interviews. Centers were asked to identify 1-2 key 
TA recipients per project at the time of each implementation interview, and 1 key TA recipient 
per project at the time of each outcomes interview. Evaluators asked the Centers to identify the 
TA recipients who were most actively involved in the projects, preferably those who had been 
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involved in initial project development if they were still available, and who were sufficiently 
knowledgeable to answer in-depth questions. TA recipient implementation interviews 
addressed questions about project development, goals, strategies, progress, challenges, and 
early outcomes, with a focus on how the respondents and their organizations experienced the 
process of working with Centers. TA recipient outcomes interviews addressed questions about 
the projects’ accomplishments and capacity-building outcomes, challenges to achieving 
outcomes, and Centers’ TA strategies that were effective. 


In 2015, the study team reached out to a total of 55 TA recipients and completed 50 interviews, 
for a response rate of 91%. In 2016, the study team conducted interviews with 37 of the 47 TA 
recipients contacted, for a response rate of 79%. In 2017, the study team reached out to a total 
of 34 TA recipients and completed 31 interviews, for a response rate of 91%. Centers could not 
identify a TA recipient for a few projects that were primarily conducted through online posting 
and dissemination of resources, without active involvement of constituents in planning the 
project or working directly with the Center. Some TA recipients who had been involved with 
project development had left the SEAs by 2016 or 2017, or had shifted responsibilities, were 
less involved in the projects, and were less available for interviews. 


Project Sample for Center and TA Recipient Interviews 
The samples of projects covered by the implementation and outcomes interviews are 
presented in Exhibits B-4 and B-5. 


Exhibit B-4. Projects Discussed in Implementation Interviews 


Projects in Center Staff Interviews Projects in TA Recipient Interviews 
(N=55) (N = 44) 

Center Type 

Regional Center 40 32 
Content Center 15 12 
Priority Area 

Teacher/leader effectiveness 35 27 
Early learning 20 17 
Year 

2015 45 39 
2016 47 39 
Both years 37 34 


Source: Center staff and TA recipient interviews, 2015 and 2016. 
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Exhibit B-5. Projects Discussed in Outcomes Interviews 


Projects in Center Staff Interviews Projects in TA Recipient Interviews 
(N = 36) (N = 31) 

Center Type 

Regional Center 25 22 
Content Center 11 9 
Priority Area 

Teacher/leader effectiveness 22 20 
Early learning 14 11 


Source: Center staff and TA recipient interviews, 2017. 


All projects discussed in outcomes interviews had previously been included in at least one of the implementation 
interviews. 


Interview Coding 

Interviews were coded and analyzed through a process of conceptual modeling, codebook 
development, piloting, and revision to achieve a minimum intercoder agreement rate of 70%. 
First, implementation interviews were transcribed from audio files and uploaded to an NVivo 
qualitative analysis software database for organization and coding. Codebook development and 
subsequent analyses were subject to these standards: 


* Clarity. Coding and analysis rules were easy to understand for members of the coding and 
analysis teams, the larger research team, and outside reviewers. 


* Consistency. Coding and analysis rules were applied in the same instances and in the 
same manner across interviews, coders, and the duration of the analysis activity. 


¢ Transparency. Coding and analysis rules were well documented and accessible to the 
coding and analysis teams, the larger research team, and outside reviewers. 


The evaluation team developed a first draft of the codes based on the research questions, the 
initial document review, the interview protocols, and team brainstorming sessions. The 
codebook was piloted and refined through an iterative process. The coding team established 
the face validity of each code by discussing the codes with the research team. Definitions for 
each code were clarified or adjusted based on the experience of pilot coding. (See detailed code 
definitions applied to interviews in Appendix C.) 


Once the codebook was ready, the process of establishing intercoder reliability occurred as 
shown in Exhibit B-6. 
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Exhibit B-6. Codebook Refinement and Intercoder Reliability Testing 


Parent Codes: 
Codebook Development * Research questions 
Lead coder develops parent codes ¢ Interview protocols 
based on research questions, interview 
protocol, and initial review of data. 


Child levels 2 & 3 Codes: 
¢ Initial read of interviews 
Validity Review and Pilot Coding 
Research team reviews codebook. 
Coding team pilots and revises 
(erele[=1 010 )0).@ 


Trial Coding 
Coding team splits into 2 pairs. Each 
pair codes a randomly selected 
interview. 


Agreement Agreement >70% 


. : ies < 70% 
Discussion & Revision a 


Coders discuss disagreements and (if 
necessary) modify problematic codes. 


Independent Coding 


' 


Spot Checks 


Every 6" interview is co-coded. 


Agreement >70%* 


*All pairs achieved 70% agreement or higher at this stage. 


During pilot coding, the team of four researchers coded two interviews as a group. Next, each 
coder coded two interviews independently, after which the team met to compare coding, 
discuss discrepancies, and further refine the codebook. Then they split the codes into two 
groups, assigning each group to a pair of coders. Throughout the coding period, each pair only 
coded their assigned group of codes. Within each pair, the members independently coded a 
randomly selected interview. If they achieved a minimum agreement score of 70% for each 
code, they began independently coding the implementation interviews, using their assigned 
half of the codebook. Coders used an agreement level template based on previous research and 
the sensitivity level of kappa scores, particularly in NVivo.”? If that level of agreement was not 


23 Bazely, P., & Jackson, K. (2013). Qualitative data analysis with NVivo. Los Angeles, CA: Sage. 
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reached, the full group of four discussed to determine whether additional codebook 
refinements were necessary. Coders met and refined the process regularly until both pairs 
achieved the desired agreement level of 70%. 


Two processes ensured that reliability was maintained throughout the coding process. First, the 
full group of coders met weekly to discuss any coding issues or anomalies. Second, after each 
member of each pair had independently coded five different interviews, the two independently 
coded one interview to check agreement levels and then met to discuss and resolve any 
discrepancies. In this process, all coding pairs’ comparisons yielded agreement rates of more 
than 70%. Most were above 80%; for four out of 26 comparisons of interview coding, 
agreement rates fell below 80% but above 70%. 


Interview results were used in the evaluation to interpret survey and activity data results, 
provide context, and provide specific project examples and illustrative quotes. When relevant, 
frequencies of specific responses were reported. However, due to the complexity and variety of 
meanings assigned by respondents to concepts such as “outcomes” and “strategies,” specific 
frequencies of interview responses reported in these analyses are more suggestive than 
conclusive. Interview questions were open-ended. Reporting the number or percentage of 
Centers that described a particular approach does not mean that the other Centers did not use 
this approach—only that staff did not report performing it when asked an open-ended 
question. Findings in this report should be considered with this caveat in mind. 


Activity Reports 


Activity reports provided a fine-grained view of Centers’ TA projects in the two priority areas. 
To collect detailed information on the individual TA activities within projects, evaluators 
designed an activity report template for the Centers to complete. The template included 
descriptors of projects and of activities nested within the projects. Fields included project goals, 
project content focus, activity dates, milestones, constituents, and product and activity types. 
Appendix C provides detail on the coding of activity types. 


At six-month intervals beginning in April 2015, evaluators sent activity report requests to the 22 
Centers, requesting that Center staff review and update the current set of projects being 
implemented in the two selected federal priority areas. Additionally, the reports collected 
information for all activities that occurred within the previous six-month period as part of these 
projects. Evaluators provided each Center director with an instruction sheet and the activity report 
template, prefilled with project information from the Center’s annual management plan, which 
was provided by the U.S. Department of Education. After receiving the completed activity reports, 
evaluators contacted Center directors to confirm and clarify any unclear or incomplete 
information. 


Activity data were collected in six cycles, covering activity from October 1, 2014, through 


September 30, 2017 (see Exhibit B-7). The evaluation team received activity reports from all 
Centers. Across the six cycles, data were collected on activities of 333 active, unique projects in 
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the two priority areas. Sample details are presented in Exhibit B-8. The activity data project 
sample is larger than the project sample used for Center staff survey analysis (see Exhibit B-8), 
since survey respondents did not report on all projects, and since the final activity data 
collection cycle occurred after completion of the final survey administration in 2017. 


Exhibit B-7. Activity Report Collection Schedule 


Activity Report Period Center Year Time of Request 
1 October 1, 2014 — March 31, 2015 3 April 2015 
2 April 1, 2015 — September 30, 2015 3 October 2015 
3 October 1, 2015 — March 31, 2016 4 April 2016 
4 April 1, 2016 — September 30, 2016 4 October 2016 
5 October 1, 2016 — March 31, 2017 5 April 2017 
6 April 1, 2017 — September 30, 2017 5 September 2017 


Exhibit B-8. Projects in Activity Reports 


Regional Content Great Teachers/ 


Total Projects Centers Centers Leaders Early Learning 


Source: Center activity reports, 2015, 2016, 2017. 


Center Documents 


Information on projects’ goals, objectives, expected outcomes, and services was collected from 
annual management plans (also called work plans) provided to the evaluation team by the U.S. 
Department of Education. This information was entered in an evaluation database, where it 
was used to prepopulate activity reports and to link with other project data when needed for 
analysis. Management plans and project documents, such as project plans, reports, and tools 
provided by the Centers, were used to prepare for surveys, to prefill project information in 
activity reports, and to clarify or supplement project information provided in interviews. 


The evaluation team reviewed the annual evaluation reports produced by the 22 Centers in late 
2016, reporting on the Centers’ fourth year. This review focused on identifying Centers’ 
outcomes measures, in order to supplement information on outcomes evidence gathered 
through interviews. 
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APPENDIX C: Evaluation Instruments and Codebooks 
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Center Staff Survey 


Thank you for taking the time to complete this survey. We are conducting this survey in order to gather 
information from Comprehensive Centers about the successes and challenges of building constituent 
capacity and providing technical assistance. 


The focus of this survey is on technical assistance in two areas: 
1. Identifying, recruiting, developing, and retaining highly effective teachers and leaders 


2. Ensuring school readiness and success of preschool-age children and their successful transition to 
kindergarten. 


We are interested in your responses as someone who provides technical assistance to state 
departments of education or to Regional Comprehensive Centers. However, there may be some 
questions on the survey that you feel do not apply to you. 


The survey should take no more than 20 minutes to complete. 


Information collected for this study falls under the confidentiality and data protection requirements of 
the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title |, Part E, Section 
183). Responses to this data collection will be used only for research purposes. For the most part, 
findings from the survey will be reported in aggregate form only and will not be associated with specific 
individuals or Centers. Quotations from individual respondents may occasionally be presented for 
illustrative purposes. We will not identify the names of these individual respondents or their Centers, 
but we may provide the broad respondent category they belong to (e.g., “SEA staff”; “Regional Center”). 
Because the number of Centers is small and because we will describe specific content and contextual 
features of the Centers’ work, in some cases it may be possible to indirectly identify some Centers. We 
will not provide information that directly identifies you to anyone outside the study team, except as 
required by law. 


This survey is voluntary. You can decide not to participate, or you can discontinue participation at any 
time without consequence. You do not have to answer any questions you do not want to, but we 


encourage you to answer as many as you can. 


Your responses will be saved as you complete the survey. If you need to come back at a later time, you 
can continue the survey where you left off. 


If you have any questions, please contact Michaela Gulemetova at 202.774.1999 extension 1956 or 
NCCEvaluation@impaqint.com. 


Thank you for your help! 


The OMB Control Number for this information collection is 1850-0914 and the expiration date is 03/31/2018. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average 20 
minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of 


information to Amy Johnson at Amy.Johnson@ed.gov. 
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1. Please select all of the projects that you worked on in any capacity in the last 12 
months. (Select all that apply) 


Yes No Don’t know 
a. Project 1 im im im 
b. Project 2 C C C 
c. Project 3 im | CO 
d. Project 4 O O O 
e. Other (please specify) im i C 


[Note: The following questions are asked for each project clicked “Yes” in question 1.] 


These questions are about your experience working on [the first selected project in question 1]. 


2. What is your role on the project? Select all that apply. 


Project and/or Center leader 
State liaison/state manager 
Consultant 

Content expert 

Data analyst 

Meeting planner or facilitator 
Professional development provider 
Project team member 
Writer/editor 

Online/social media developer 
Quality assurance reviewer 
Advisor/consultant 

Evaluator 

Other 


ay a (7 I pt Gf 


If you indicated that you have an "Other" role, please describe the role: 
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3. Which of these factors have been the most significant challenges in this project? Select 
up to three. 
[Allow only 3 selections] 
. Turnover in chief state school officer and/or other SEA leadership 
. Turnover among state education agency (SEA) staff 
. Turnover among Center staff 
. Changing priorities at an SEA 
. Mismatch between stated SEA needs and Center federal priorities 
Difficulty establishing relationships with an SEA 
. Policy shifts in a state government 
. Cuts in SEA staffing/budgets 
Meeting diverse needs across SEAs (and/or regions) 
Lack of communication within an SEA 
. Lack of Center financial resources 
|. Lack of access to appropriate expertise 
m. Project timeline constraints 
n. Other 


Aa rma nmroaQqao»#w 
Py a a a Fe UB 


If you indicated an “Other” challenge, please describe the challenge: 


4. Which of these have been most significant in supporting success in this project? Select up 
to three. 
[Allow only 3 selections] 
. Content expertise among Center staff 
. Knowledge of state context among Center staff 
. Expertise among SEA staff 
. Supportive leadership in Center 
. SEA Leadership commitment to the project 
Strong relationships between Center staff and SEA leadership 
. Strong relationships between Center staff and SEA staff 
. Collaboration with Content Comprehensive Center 
Collaboration with Regional Comprehensive Center 
Collaboration with a Regional Educational Laboratory 
. Collaboration with an institution of higher education 
Collaboration with a government agency 
.Collaboration with a professional association (e.g., ASCD, CCSSO) 
. Collaboration with a subcontractor 
. Other 


307 AT aw moaQanqnqen»sorwno 
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If you indicated an “Other” supporting factor, please describe it: 


5. Who are your collaborators on this project? (specific Comprehensive Centers, Regional 
Educational Laboratories, institutions of higher education, etc.) 


6. To what extent is this project succeeding (or has succeeded) in building the capacity of 
your constituents (SEA staff or other participants)? 
©) Toa great extent 
© Toa moderate extent 
©) To some extent 
© Not at all 


Please explain your response: 


{GO TO next project and ask questions 2-6, until all projects have appeared. Then ask the 
following question once.} 


7. Of the projects you indicated earlier that you worked on, is there one that would be suitable 
for a project profile (an in-depth description of a project), in that it is high-leverage (is likely 
to lead to extensive capacity building) and long-term (longer than one year)? In what ways 
is it likely to be high-leverage? 
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Center Staff Survey Codes for Open-Ended Question 


Survey 


Item Code Definition Example 


6. To what extent is this project succeeding (or has succeeded) in building the capacity of your constituents? 
Please explain your response: 
6 Project progress TA provided/needs addressed/some SEA staff asked for support with tight turn- 
examples of capacity change around and we provided valuable feedback 
on their planning. 
6 Don’t know or difficult to | Difficult for Center respondent to assess Hard to measure impact with this project 
measure or measure capacity-building (CB) in a 
formal way; or Center staff respondent 
says they don't know the extent to which 
CB was built 
6 Implementation Various challenges have limited the Maintaining membership has been a 
challenges extent to which CB has been built - challenge so capacity building is slow; ESSA 
includes SEA capacity, SEA turnover, has taken priority; SEA has limited staff 
other federal/ SEA priorities 
6 Work still in progress/not | project is still in progress; response The Summit was designed to be a ‘kick off' 
widely used yet specifically indicates this; work is event to launch additional regional 
intended to be more widely used meetings and resources over time. Those 
follow through activities are now under 
way and continue to build knowledge 
about professional learning standards 
6 Light touch: not meant to | Center's work was not intended to These were informational and used as the 
extensively build capacity | extensively build SEA capacity; Center launch for our materials. 
just created a product or for various 
reasons, only conducted limited TA/ CB 
6 Outcome Explanation for rating is based on some This is my best guess based on the 
measure/feedback cited sort of feedback or evidence from feedback | received. The events are well- 
constituent received but | can only guess at building 
capacity based on the conversations 
6 Variable outcomes Outcomes varied by state or constituent | This project includes two states and one 
state has been greatly involved and the 
second state not as much. Changing 
priorities, including signing of ESSA, put the 
project on hold in one state. 
6 No explanation Rating provided, but 
response/explanation is blank, or vague 
statements cannot be coded. 
6 No rating no rating provided 
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Technical Assistance Recipient Survey 


Thank you for taking the time to complete this survey. The US Department of Education supports 22 
Comprehensive Technical Assistance Centers to help increase state education agencies’ capacity to 
assist districts and schools in meeting student achievement goals. As part of an external evaluation of 
the national Comprehensive Centers program, IMPAQ International is surveying recipients of technical 
assistance like you about your experience with the program and related capacity-building outcomes. 
Your responses will help assess outcomes and inform improvement of the Comprehensive Centers 
program. 


The focus of this survey is on technical assistance in two areas: 
1. Identifying, recruiting, developing, and retaining highly effective teachers and leaders 


2. Ensuring school readiness and success of preschool-age children and their successful transition 
to kindergarten. 


The survey should take no more than 15 minutes to complete. 


Information collected for this study falls under the confidentiality and data protection requirements of 
the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title |, Part E, Section 
183). Responses to this data collection will be used only for research purposes. For the most part, 
findings from the survey will be reported in aggregate form only and will not be associated with specific 
individuals or Centers. Quotations from individual respondents may occasionally be presented for 
illustrative purposes. We will not identify the names of these individual respondents or their Centers, 
but we may provide the broad respondent category they belong to (e.g., “SEA staff”; “Regional Center”). 
Because the number of Centers is small and because we will describe specific content and contextual 
features of the Centers’ work, in some cases it may be possible to indirectly identify some Centers. We 
will not provide information that directly identifies you to anyone outside the study team, except as 
required by law. 


This survey is voluntary. You can decide not to participate, or you can discontinue participation at any 
time without consequence. You do not have to answer any questions you do not want to, but we 


encourage you to answer as many as you can. 


Your responses will be saved as you complete the survey. If you need to come back at a later time, you 
can continue the survey where you left off. 


If you have any questions, please contact Michaela Gulemetova at 202.774.1999 extension 1956 or 
NCCEvaluation@impaqint.com. 


Thank you for your help! 


The OMB Control Number for this information collection is 1850-0914 and the expiration date is 03/31/2018. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average 15 
minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of 
information to Amy Johnson at Amy.Johnson@ed.gov 
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1. To what extent do you agree with the following statements about your relationship with 
the [Center]? (Select ONE response in each row) 


Neither 
Strongly Disagree Strongly 


Disagree Disagree Nor Agree Agree Agree 


a. We have frequent 
communication. 

b. Our relationship is 
collaborative. 

c. They understand my 
organization’s needs. 

d. They provide technical 
assistance in timely manner. 


O O O O O 
O O O O O 
O O O O O 


Next, we will ask you questions about specific technical assistance (TA) you have received 
during the last 12 months in the Department of Education’s priority areas related to: 


e Identifying, recruiting, developing, and retaining highly effective teachers and leaders 
(great teachers and leaders) and 

e Ensuring the school readiness and success of preschool-age children and their successful 
transition to kindergarten (early childhood learning) 


2. Have you personally received the any of following specific TA from [Center or Centers] 
during the last 12 months? (Please select a response for each project) 


Yes To) Don’t know 
a. Project 1 O O O 
b. Project 2 O O O 
c. Project 3 O O O 
d. Project 4 (> O O 
e. Other (Please specify) O O O 


The rest of the survey will ask you about your experiences with the specific TA you indicated 
you received in the prior question. 
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3. To what extent do you agree with the following statements about the technical assistance 
(TA) you received from [Center] during the last 12 months? (Select ONE response in each 


row) 
Neither 
Strongly Disagree Strongly 
Disagree Disagree Nor Agree Agree Agree 
a. Overall, the TA content 
was of high quality. O O O O O 
b. Overall, the TA delivery 
was of high quality. O O O O O 
c. Overall, the TA was 
relevant to my work. O O O O O 
d. Overall, the TA was 
O O O O O 


useful. 


Please answer the following questions for the TA you indicated you received at the beginning of 

this survey. 

4. After receiving technical assistance last year, have you engaged in any of the following 
activities? (Check ALL that apply) 


© a. Reviewed the materials/resources/handouts provided by the [Center] 

© b. Looked up references provided by the [Center] 

©) c. Researched additional materials on the topic 

© d. Used the information in drafting internal memos/plans/reports 

© e. Used the information in developing new processes/projects/programs 

©) f. Used the information to overcome a barrier or challenge 

© g. Made recommendations to SEA leadership or staff 

© h. Other 

©) i. Ihave not yet engaged in any activities related to the TA, but have made plans to do so 
© j. No, | have not engaged in any activities related to the TA | received from the [Center] 


If you indicated an "Other" activity, please describe the activity: 


5. Have you discussed the information you received through the TA with any of the 
following? (Check ALL that apply) 
© a. [Center] staff/event organizers 

. Other technical assistance recipients from outside our organization (e.g., other SEAs) 

. LEA staff 

. Other staff within your organization 

. Other 

. No, | have not discussed the information 


OOO0O° 


If you indicated that you discussed the information with an "Other" person, please 
describe who that was: 
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6. To what extent do you agree that the technical assistance you received during last 12 
months helped you in the following ways? (Select ONE response in each row) 


To what extent did the TA you To some To a great Not 
received help you.... Notatall Very little extent extent PVoye) [Cer-]o) (<] 
a. Solve a problem? O O O O O 
b. Access the information 
you wanted? O O O O O 
c. Increase your knowledge 
and expertise in the CG) 'S O O O 
topic? 
d. Identify additional 
training events, webinars, Cy 'S C) O (Ss 


and seminars of interest? 
e. Build collaborations with 
professional experts 
outside your O O O O O 
organization? 
f. Access useful information 
or ideas from other O O O O O 
states? 
g. Learn about or obtain 
new tools that you can 
use in your work? 
h. Learn new strategies that 
you Can use in your work? 
i. Perform your jobina 
more informed and 
efficient way? 
j. Other (Please 
specify): 
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7. To what extent do you agree that the technical assistance you received during the last 12 
months helped your organization in the following ways? (Select ONE response in each row) 


To what extent did the TA you Very Tosome  Toagreat Not 
received help your organization... Not at all little extent extent Vole) |(er=] 0) (<1 
a. Develop and implement 
internal staff trainings? O O O o O 
b. Incorporate the new 
information into internal O O > O O 
procedures and processes? 
c. Redesign state agency 
structures, teams and O O O O O 
departments? 
d. Improve communication 
channels across state 
agencies? 
e. Improve communication 
channels with LEAs? 
f. Partner more with external 
experts? O O O O O 
g. Shape new and existing 
programs and policies in the 
area of great teachers and O O O O O 
leaders? 
h. Provide better support to 
LEAs in the area of great O © O ee, O 
teachers and leaders? 
i. Shape new and existing 
programs and policies in the O O O O ©) 
area of early learning? 
j. Provide better support to 
LEAs in the area of early O O O 'S O 
learning? 
k. Identify other needs for 


information? O O O O O 
|. Other: ‘fs O O O O 


8. Do you have specific plans to use the information you received from the [Center] in the 
upcoming 12 months? (Select ONE) 


© a. Yes > if yes, please elaborate how: 


© b. No 
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9. Has the [Center]’s technical assistance enabled you to accomplish what you may not have 
otherwise accomplished? 


© a. Yes > Please explain: 


© b. No 


10. Please describe the aspects of the services and/or resources that have been most helpful 
and why: 


11. Please describe the aspects of the services and/or resources that have been least helpful 
and why: 


12. Please describe if you have any needs that were not met by the services and/or resources: 


13. What challenges have you encountered in your experience with the [Center] during the 
last 12 months? (Check ALL that apply) 


© a. Lack of understanding of our needs by [Center] staff 

© b. Delay in receiving needed TA 

© c. Staff turnover at our organization 

© d. Staff turnover at the [Center] 

© e. Difficulty scheduling time for our staff to participate 

© f. Difficulty in communication with the [Center] 

© g. Lack of skills/expertise by [Center] staff to address our needs 

© h. Lack of alignment between my organization’s needs and [Center] priorities 
©) i. Insufficient number of in-person meetings with the [Center] 

© j. Unclear understanding of the role of the [Center] and what it can offer 
© k. No challenges 

© |. Other 


If you indicated that you have had "Other" challenges. Please describe the challenge. 


14. Please provide any additional comments about your experience with the 
[Center]: 


15. Please identify your organization: (Select ONE) 
© a. State Education Agency 
© b. Local Education Agency/School District 
© c. School 
©) d. Comprehensive Center 
© e. Other (please specify) 
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If you indicated an "Other" organization, please describe your organization: 


16. What is your role in your organization? 


17. How long have you been in your current role? (Select ONE) 
© a.0-1 years 
© b. 2-3 years 
© c.4or more years 


18. How long have you been with your organization? (Select ONE) 
© a.0-1 years 
© b.2-3 years 
© c.4-10 years 
© d.11 or more years 


Thank you! 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program C-13 


Technical Assistance Recipient Survey Codebook for Open-Ended Survey Questions 


Survey 
Item 


(efoyo l=) 


Definition 


Examples 


10. Please describe the aspects of the services and/or resources that have been most helpful and why. 


10 ~=|Support for Plan or Center supports SEA in developing plans, reports, other Assistance in development of professional learning standards, 
Product Creation documents. teacher equity plans, strategic plans 
10 ~=—‘|Access to Experts References to Centers’ connections to outside experts or to |Brought in expert presenter or panel member, provided 
expertise of the Center staff. expert review 
10‘ |Provisions of Products/ Center provides product or resource created by Center or Specific Center product provides the LEAs with a doable 
Materials/ Resources available through Center partner, resource library or scan format of school improvement 
10 _——(|Training Center provides or supports training for constituents Specific Center staff member's presentation to the leadership 
Fellows - very useful 
10  |Cross-State and Peer References to peer group contacts made through the Finding out how other states address problems relating to 
Sharing Center, usually in other states teacher evaluation, especially for special education teachers 
and other teachers without the traditional classroom 
assignment. 
10 ~=|Thought Partner References to helping SEA think through issues, acting asa |Mostly specific Center staff member's readiness to be a 
critical friend thought partner and talk to us about ways his center could 
support us. 
10  |Stakeholder References to Center's role in reaching out, obtaining Her facilitator skills were so greatly appreciated by me. 
Engagement/Facilitation {stakeholder buy-in, including meeting facilitation Together we were much better in soliciting responses from 
the stakeholders and gathering feedback from them. 
10 ~=|Relationships References to the importance of the relationship between |They have been true partners and a great resource in this 
the Center and the SEA work. 
10 |Responsiveness References to Center's responsiveness to SEA needs or Center has been very responsive to our requests, questions, 
requests and suggestions. 
11. Please describe the aspects of the services and/or resources that have been least helpful and why. 
11. |All helpful Respondent states that all services were helpful or can’t Cannot recall a service or resource that isn’t helpful 
identify any that were unhelpful 
11.  |Relevance or quality Deficiency or limitations of a product or service is noted. Certain resources “have not been as pertinent in our region”; 
concerns “At times the protocols were a bit redundant and, therefore, 
tedious to follow.” “Webinars are not convenient.” 
11.  |Availability Respondent would have liked more time from Center staff, |“More side by side assistance”; “Sometimes difficult to get 
more accessibility, or more time on-site with the SEA staff |hold of.” 
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Survey 
Item 


Code 


Definition 


Examples 


11.  |SEA issues. Limitations in using the TA related to the SEA, not the Things change rapidly so not having the capacity on staff (as 
Center opposed to outside TA) is not always timely - through no fault 
of the TA. At some point, state agencies just need to be 
staffed appropriately for reform work. 
12. Please describe if you have any needs that were not met by the services and/or resources. 
12. |All needs were met Respondent states that all needs were met by Center or “They provided all support that we need.” 
can’t think of any other needs they wanted the Center to 
address 
12. |More TA Would like more resources, connections, staff availability or |“It is often difficult to book our support folks from the center 
TA in some priority areas for in-person workshops when we need them as their 
calendars are full.” 
“| simply would love to have another year of assistance! 
Never enough time to learn all we need to learn! Great 
experience!” 
12. |Center staff turnover Center staff changes caused confusion/would have liked “Consistency in the group” 
more clarity or consistency 
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Design-Focused Interview Protocol for Center Staff 


The OMB Control Number for this information collection is 1850-0935 and the expiration date is 03/31/2018. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average 60 
minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of 


information to Amy Johnson at Amy.Johnson@ed.gov . 


[NOTE TO INTERVIEWER: Below are suggested introductory remarks. While it is not necessary to 
follow this as a script, it is important that you cover all of the main points contained here.] 


| work for IMPAQ International, and we are evaluating the system of Comprehensive Technical 
Assistance Centers. The U.S. Department of Education’s Institute for Education Sciences 
contracted with us to conduct this study. You may recall that we visited last year to learn about 
your Center. 


As part of our study, we will be interviewing directors, managers, and staff at the Centers to 
understand the work that you do. The purpose of the study is to gather data to describe the work 
of the Centers and how the Centers build the capacities of state departments of education, and 
to report that information to the Department of Education. What you have to say is important to 
us and we appreciate you helping us understand your work. We want to assure you that 
participation in this interview is voluntary. We estimate that this interview will take 60 minutes. 


[NOTE TO INTERVIEWER: Please read the following to the respondent(s): 


“Information collected for this study comes under the confidentiality and data protection 
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, 
Title |, Part E, Section 183). Responses to this data collection will be used only for research 
purposes. Findings from the interview data will be reported in summary form and individuals will 
not be identified by name. However, respondents’ roles and the CC they work with may be 
identified in the report, which may lead to individuals’ being identified. Other than this situation 
that we have made respondents aware of, we will not provide information that identifies you to 
anyone outside the study team, except as required by law.” 


Before we begin, do you have any questions about the purpose of the evaluation or our 
confidentiality policy? 


To start, do you mind if we audio record our session for the accuracy of our note taking? 
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These questions will be asked of Center leadership (manager or director) and may include a larger 
group including technical assistance providers. If respondents in a group interview differ in their 
responses/opinion, those differences will be recorded in the interviewer notes. Prior to 
conducting the interview, the interviewer will review relevant extant documents such as 
proposals, annual performance review reports, evaluation reports, etc. Some of the answers to 
questions asked below may be in those documents. If so, the interviewer will start the question 
by referring to information gleaned from the document, and will ask the respondent to verify the 
accuracy of the information. 


Capacity Building Definition 

1. For the purposes of your Center work with state education agencies (SEAs) [or for Content 
Centers, SEAs and Regional Centers], how do you define capacity building? What is the basis 
for your definition (research, experience, other Centers)? 


PROBES: 

How do you talk about capacity building among your Center staff? 

Do you focus on different types of capacity building (e.g. human, organizational, structural, 
etc.? If so, please tell us about that. 

[For Content Center interviews: Do you focus on different types of capacity building for 
Regional Centers as compared to SEAs? Does capacity building look different for Regional 
Centers and SEAs?] 

How do you discuss capacity building with your SEA constituents—to what extent do you use 
the term “capacity building”? 


2. When thinking about your work, how do you know when you have built capacity? What does 
that “look like”? In other words, what observed changes in stakeholder behaviors indicate 
capacity building? What are the outcomes that you expect to see? 


3. Do you collect evidence to assess your progress in building capacity? If so, what kinds of 
evidence do you collect? 


Theory of Action 

We are interested in the assumptions behind your work with SEAs—how you decide, out of all of 
the options available, what work you will do with the SEAs (and for Content Centers, the Regional 
Centers) and how you monitor and adjust that work as needed. Please describe (and/or show us 
graphically) your theory of change and explain how you arrived at it. 


4. How do you assess SEA needs—what information do you request, collect and review? 


PROBES: 

Who is involved in conversations/meetings about SEA goals? [for Content Centers, ask also 
about Regional Center goals and goals that might be generated through means other than 
directly by states or Regional Centers, e.g., addressing a broad, multi-sector or national need] 
To what extent do your SEAs present you with their goals, and to what extent do you suggest 
goal options to them? 
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10. 


11. 


How do you determine the needs you can help the SEA meet or the goals the SEA can pursue 
through collaboration with your Center? 


How do you assess the SEA’s current level of capacity to meet their goals? [For Content Center 
interviews, also ask: is the process different in your work with Regional Centers? If yes, in 
what ways?] 


PROBES: 

To what extent do your SEAs present you with their goals, and to what extent do you suggest 
goal options to them? 

If different SEA personnel present you with different goals, how do you resolve this? 


To what extent do the states’ identified needs align with the Federal priority areas? Have you 
encountered many state needs or requests for TA that did not align with these priorities? Or 
that you could not address for some other reason? If so, please explain. How do you respond 
in these cases? 


Once SEAs have determined their goals, how do you choose your Center objectives—the 
overall results your Center hopes to achieve to support the SEA’s goals? 


PROBES: 

Who is involved in conversations/meetings about Center objectives? 

To what extent are the Center objectives determined by the SEA’s requests of you, and to what 
extent do you suggest the Center’s objectives to the SEA? 


How do you choose which specific strategies or projects to pursue to achieve your objectives? 
What is your theory about how these strategies, activities, or projects will contribute to 
achieving your objectives? 


PROBES: 

Who is involved in conversations/meetings about strategies? 
What do you take into consideration when deciding upon strategies (feasibility, evidence, 
availability of expertise, experience in other projects, expectations of SEA, likelihood of 
building capacity, etc.)? 
To what extent are your projects/strategies determined by the SEAs’ requests of you, and to 
what extent do you suggest the projects to the SEAs? 


How do you customize strategies to the SEA’s existing capacity level and needs? 


Can you describe how the local context—political, economic, etc.—affects your choice of 
strategies/projects, if at all? 


PROBES: 
How do you become aware of these contextual factors? 
How influential do you think context is in your decision making? 
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12. 


How does the availability of Center resources (financial, human, etc.) enter into your choice 
of strategies? [/f resources do affect strategy choice: If you’re making decisions about projects 
based on resources, do you involve stakeholders (SEAs/Rocs) in the decision process? If so, 
how? ] 


The next set of questions asks about practical aspects of your project work, such as how you 
monitor project activities and how you determine timeframes. These questions will help us 
describe the process by which you do your work, and how your theories of actions inform the 
process. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


How do you track and assess progress in your projects? What evidence do you collect or 
review to do this? 


How do you choose which outcomes to track for specific projects or the Center as a whole? 
Can you give us any examples of outcomes that you’ve documented so far? 


PROBES: 

Who is involved in decisions about outcomes? 

What implementation milestones do you track, and how? 

What do you do if implementation milestones aren’t met? 

How do you decide on longer-term outcomes for your Center work? 
How do you measure your longer-term outcomes? 


How do you determine the timeframe for a particular project? Do you have any rules of 
thumb about how long projects should be? 


PROBES: 
How do you take stakeholder expectations and availability into account when developing 
timelines? How does the SEAs’ planning process impact project timelines? 


To what extent, if any, has your Center updated plans during a project? If so, how did you 
become aware of a need to change something? Can you describe this process of changing 


projects or strategies? How often does this happen? 


As you track progress toward your Center objectives, to what extent do you also assess SEAs’ 
progress toward their goals? 


Have you modified your theory of change over time? If so, please describe how and why. 


What else would you like to share about your approach to working with SEAs [or RCs, for 
Content Centers]? 
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Implementation-Focused Interview Protocol for Center Staff 


The OMB Control Number for this information collection is 1850-0935> and the expiration date is 03/31/2018. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average one 
to two hours per response. Send comments regarding this burden estimate or any other aspect of this 


collection of information to Amy.Johnson@ed.gov 


[NOTE TO INTERVIEWER: Below are suggested introductory remarks. While it is not necessary to 
follow this as a script, it is important that you cover all of the main points contained here.] 


| work for IMPAQ International, and we are researching the Comprehensive Technical Assistance 
Centers. The U.S. Department of Education’s Institute for Education Sciences contracted with us 
to conduct this study. You may recall that we visited last year to learn about the Center. 


As part of our study, we will be interviewing directors, managers, and staff at the Centers to 
understand the work that you do. The purpose of the study is to gather data to describe the 
work of the Centers and how the Centers build the capacities of state departments of education, 
and to report that information to the Department of Education. What you have to say is 
important to us and we appreciate you helping us understand your work. We want to assure 
you that participation in this interview is voluntary. We estimate that this interview will take 60 
minutes. 


[NOTE TO INTERVIEWER: Please read the following to the respondent(s): 


“Information collected for this study comes under the confidentiality and data protection 
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, 
Title |, Part E, Section 183). Responses to this data collection will be used only for research 
purposes. Findings from the interview data will be reported in summary form and individuals 
will not be identified by name. However, respondents’ roles and the CC they work with may be 
identified in the report, which may lead to individuals’ being identified. Other than this situation 
that we have made respondents aware of, we will not provide information that identifies you to 
anyone outside the study team, except as required by law.” 


Before we begin, do you have any questions about the purpose of the evaluation or our 
confidentiality policy? 


To start, do you mind if we audio record our session for the accuracy of our note taking? 
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All questions will be asked about the Early Learning priority area first, and then about the Great 
Teachers/Leaders priority area. Questions are for staff of both Regional Centers and Content 
Centers unless specifically noted otherwise. Interview respondents for the first section (“Overall 
Perspective”) should be Center Directors, Content Specialists in the Early Learning or 
Great/Teachers Leaders areas, and lead TA providers who work within these priority areas. 
Interview respondents for the second section (“Project Background”) should be lead TA 
providers who work on the identified projects, but Center Directors and Content Specialists 
may also be involved with these projects and should be included as relevant. Prior to 
conducting the interview, the interviewer will review relevant extant documents such as 
proposals, annual performance review reports, evaluation reports, etc. Some of the answers 
to questions asked below may be in those documents. If so, the interviewer will start the 
question by referring to information gleaned from the document, and will ask the respondent 
to verify the accuracy of the information. 


Further introduction from the Interviewer: We are going to ask you questions that focus on 
two priority areas. We will begin with the Early Learning priority area; we will first ask about 
your overall approach to TA in this area, and then we will ask about specific projects. We will 
then ask about the Great Teachers/Leaders priority area, again starting with questions about 
your overall approach followed by questions about specific projects. [Note to interviewer; ask 
ALL questions, including both overall perspective and project-specific questions, first about 
Early Learning and then repeat all the questions focusing on Great /Teachers Leaders.] 


Overall Perspective on the Priority Area 
Ask Regional Centers Only: 


1. Are there needs for technical assistance within the Early Learning [Great Teachers/ Leaders] 
priority area that have been identified by your states? If so, please describe the needs. 


2. If there are needs, how did they come to your attention? If this area has not been identified 
as a priority need, do you know why not? What do SEA staff say about why this area is or is 
not a priority TA need? 


PROBE: What evidence have you reviewed that helps you understand the needs? 


3. Is there a range of early learning [great teachers/leaders] TA needs across states in your 
region? How do states in your region differ in the types and level of these needs? 


Ask Content Centers Only: 

4. Are there needs (whether state, regional, national needs) in this priority area that have 
fallen within the purview of your center, or intersected with the focus of your center? If so 
please describe these needs. How did these needs come to your attention? 
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PROBES: Have early learning [great teachers/ leaders] needs come up in discussions with 
Regional Centers, other Content Centers, SEAs, or other constituents? What evidence have 
you reviewed that helps you understand the needs? 


5. Have you encountered a range of early learning [great teachers/leaders] needs across states 
and regions? If so please describe the differences in level or types of needs. 


If no needs have been identified in the Early Learning priority area, skip the remaining 
questions for Early Learning and start over with the Great Teachers/Leaders area. 


6. Have the needs in this priority area changed since first identified to the Center? If so, please 
describe how. How did you learn about any changes? 


7. Who (individuals, agencies, groups) has been part of the Center’s discussion of early 
learning [great teachers/leaders] needs and any plans to address them? 


PROBES: 

e Chief State School Officers or other SEA leadership 

e Other SEA staff (such as middle management staff) 

e Leadership in other Agencies such as Human Services, Family and Children's Services, etc. 
e Other staff within these agencies (such as middle management staff) 

e Others such as governors’ offices, task forces, etc. 

e Advocacy or TA groups 

e Other Centers, whether Regional or Content Centers 


8. Do you have designated staff and partners who work with you on early learning [great 
teachers/leaders] projects? [If so], please identify them and their expertise relevant to this 
area of work. 


9. Do you have a theory of action that is specific to early learning? [If so], please describe it. By 
theory of action, we mean how your Center works to build SEA capacity in early learning, 
including the chain of causal assumptions linking program resources, activities, intermediate 
outcomes and ultimate goals. Specifically, we’d like to know: 


9a. What are you trying to help the states achieve in early learning? What are the expected 
outcomes? How is this TOA anchored to the needs we discussed above? 


9b. Can you identify any overall approaches or strategies (e.g., ones that cut across specific 
projects or activities) that you use to help states achieve these outcomes in early 
learning [great teachers/leaders]—approaches or strategies that are specific to this 
priority area? How do the strategies lead to the outcomes? 
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10. 


11. 


12. 


13. 


14. 


[Note to interviewer: whether the Center being interviewed is a Regional or Content Center, 
be sure to ask about how they work with both types of Centers.] Do [other] Regional 
Centers work with you on these approaches or strategies? If so, how? Do [other] Content 
Centers work with you on these approaches or strategies? If so, how? 


What has gone particularly well in your work with other Centers—with both Regional and 
Content Centers-- in this priority area? What has been challenging in working with other 
Centers—with both Regional and Content Centers? 


[Note to interviewer: ask this of both types of Centers, since Content Centers may receive 
training from other Content Centers.] Have you received TA or training in this priority area 
from any of the [other] Content Centers (directed to your own center as distinct from or in 
addition to SEA staff)? If so, please describe and tell me how you have used this 
information. 


As you think about all of your work in early learning [great teachers/leaders], what do you 
consider to be going very well? How do you know? What indicators do you rely on? 


As you think about all of your work in early learning [great teachers/leaders], what have you 
found to be unsuccessful or challenging? How do you know? What indicators do you rely 
on? 


Project Implementation 


We will now discuss up to 2 projects from your center as examples of how the Center works in 
this priority area. As we discussed with you earlier when planning this visit, if you have more 
than two projects in this priority area, please select the two projects that you consider the best 
illustrations of your work in this area in terms of their capacity building potential, or uniqueness 
to the work that you do at your center. 


15. 


16. 


17. 


18. 


[Start with one project] What do you call this project? 


Is this a state, regional, or national project? Please identify the specific states or regions, if 
relevant, or describe the scope of the project. 


Is the project completed or ongoing? What was the start date, and the ending date (or 
anticipated end date) if applicable? 


Why did you select this project as a focal project for discussion? 
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19. 


20. 


21. 


22. 


23. 


What prompted this project? Who initiated the concept of this project? Who participated in 
the early discussions? Who helped plan the project? What needs were identified as having 
the greatest priority? How did you decide on priorities? 


PROBES: 

e Chief State School Officers 

e SEA staff below leadership level 
e Other Centers 

e Other partner organizations 


Which constituents (SEA representatives or others) do you work with most closely on this 
project? Please describe their roles on the project. Can you give us their contact information 
so we can contact them for interviews to obtain the TA recipient perspective? 


Please identify and describe the roles of any organizations serving as partners in this 
project. What is their role in implementing the project and how do the partners work 
together? 


PROBES: 

Other Regional Centers or Content Centers 
RELs 

Other TA providers or experts 


Which Center staff members work on this project? Please describe their respective roles on 
the project. 


What were the elements of the context —political, fiscal, legal, economic, etc.—that were 
influential factors during the project’s early development? Did these elements facilitate or 
impede the project in any way? If so, how? 


PROBES: 

How influential do you think these elements of the context were? 

How did it impact what your team was thinking and doing in relation to this project? 
Did the state(s) have any Federal or state grant money behind this project? 


Project Goals 


24. 


25. 


What are the broad goals of the project? What areas of SEA capacity are expected to 
improve? Which SEA staff are the primary targets of this project? 


PROBES: 

Human capacity/knowledge/skills 
Organizational capacity 
Structural capacity/policy change 


Have the goals of the project changed over time? If so, please describe. 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program C-24 


26. What are the key strategies and activities—including training events, products, meetings— 
that are part of achieving the goals? Who are the recipients of these strategies and 
activities? 


27. Have you modified the strategies or activities originally planned for this project? [If so] 
Please tell us about how and why these were changed. 


Project Communication 


28. How often do you meet or talk with SEA representatives, or other key constituents, if 
relevant, regarding planning and implementation of the project? With which SEA 
representatives do you meet? What do you discuss? How do these discussions inform 
implementation of the project? 


Project Implementation and Outcomes 


29. What is your definition of success in this project? What will we ‘see’ if this project is 
successful? What are the milestones for the project? 


30. Based on your definitions of a successful project, how have things gone so far? What is 
your evidence for making this assessment of progress? 


PROBES: 
Has the project met its milestones so far? Why/why not? 


31. What have been the outcomes so far? Have you seen changes in the capacity of the SEAs or 
other targeted constituencies? Please describe any changes in each of these types of 
capacity, and the evidence of the change: 

e Knowledge and skills (e.g., knowledge, expertise, skills, will, understanding necessary to 
implement something) 

e Organizational (e.g., interaction, collaboration, communication among individuals in a 
system; evaluation and feedback on processes and structures for continuous 
improvement) 

e Structural/policy change (e.g., policies, procedures, and formalized practices) PROBE: 
Has the SEA capacity to work with LEAs improved? 


32. What has been the most challenging aspect of this project—including any challenges to 
implementation or to building SEA capacity? Please describe. How have you handled the 
challenges? 


PROBES: 

Has there been turnover within the SEA? 

Have there been challenges in communicating with the SEA? 
Have there been other challenges in working with the SEA 
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33. 


34. 


35. 


36. 


37. 


Have there been challenges in working with partners? 

What mokes it difficult to build capacity? 

Overall what do you think has been the most successful aspect of this project? Please 
describe it. To what do you attribute this success? What have you learned from this? 


Are there plans for the Center to build on, replicate, or continue this project? 


Have the SEAs or other constituents begun to take over or "own" this work in any way? 
What are their next steps? 


Are there upcoming events associated with this project that we may be able to observe to 
help us understand project implementation? //f so] Please describe them and provide dates. 


Are there products/materials associated with this project (in addition to those we’ve 
received prior to the interview) that we could collect and read to help us understand project 
implementation or outcomes? 


Repeat project questions for the 2” project in the Early Learning area, if applicable. Then repeat 
all questions for the Great Teachers/Leaders priority area. 
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Implementation-Focused Interview Protocol for TA Recipients 


The OMB Control Number for this information collection is 1850-0935 and the expiration date is 03/31/2018. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average 60 
minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of 


information to Amy.Johnson@ed.gov. 


[NOTE TO INTERVIEWER: Below are suggested introductory remarks. While it is not necessary to 
follow this as a script, it is important that you cover all of the main points contained here.] 


| work for IMPAQ International, and we are evaluating the system of Comprehensive Technical 
Assistance Centers. The U.S. Department of Education’s Institute for Education Sciences 
contracted with us to conduct this study. 


As part of our study, we will be interviewing recipients of Center work. The purpose of the study 
is to gather data to describe the work of the Centers and how the Centers build the capacities of 
state departments of education. We will use the information we learn from our visit today to 
report our results to the U.S. Department of Education. What you have to say is important to us 
and we appreciate your helping us understand the Centers’ work. We want to assure you that 
participation in this interview is voluntary. We estimate that this interview will take 60 minutes. 


[NOTE TO INTERVIEWER: Please read the following to the respondent(s): 


“Information collected for this study comes under the confidentiality and data protection 
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, 
Title |, Part E, Section 183). Responses to this data collection will be used only for research 
purposes. The reports prepared for this study will include information that is summarized and 
aggregated and should not associate responses with a specific Center, state, district or 
individual. Findings from the interview data will be reported in summary form and individuals 
will not be identified by name. However, respondents’ roles and the CC they work with may be 
identified in the report, which may lead to individuals’ being identified. Other than this situation 
that we have made respondents aware of, we will not provide information that identifies you to 
anyone outside the study team, except as required by law.” 


Before we begin, do you have any questions about the purpose of the evaluation or our 
confidentiality policy? 


To start, do you mind if we audio record our session for the accuracy of my note taking? 
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Individual Information and Knowledge of Centers 


1. 


10. 


Please describe your position and your responsibilities with the SEA. 
How long have you been in this position? How long have you worked with the SEA? 


Please describe any other background particularly relevant to your current work with the 
SEAs? 


When did you first work with or receive services from any of the Comprehensive Centers? 
Which Centers have you worked with or received technical assistance from since that time? 


| would like to focus now on the Centers that are currently funded (funded Centers since 
late 2012). We’d like to hear about ways you've worked with or received services from 
these Centers. How did the Center try to learn about your needs? 


PROBES: 

What data or documents did the Center request? 

Whom did the Center interview? 

What was discussed in the early contacts/meetings? 

Have you had regular contact with them since the first contact? 
Is this typically in person, by phone, or by e-mail? 


Please describe the major projects or services the Center has provided you or that you've 
participated in with the Center(s). 


How are decisions made about what services the Center should provide to the SEA? 


How would you characterize the specific role of the Center(s) in serving the SEAs, as distinct 
from other TA providers available to the SEA? 


What are you most hoping to get from the Center? What needs do you hope the Center will 
address? 


PROBE: 
In what ways do you hope the Center will build your capacity? 


Do you believe the Centers have been effective in addressing the needs of the SEA and the 
state? If yes, please describe how and what evidence you have of their effectiveness. If no, 
please explain why and how/in what ways they have not been effective in serving the 
SEA/state needs. 
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Project Background 


Now we'd like to discuss one or two specific projects that the Center staff has identified as 
illustrative that they are implementing with your SEA: 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


How would you briefly describe the project to someone who isn’t 
familiar with it? 


What are the key strategies and activities —including training events, products, meetings— 
that are part of achieving the goals? 


How does the project fit into the overall work at the SEA or your division? 
PROBES: 

Are there other initiatives or projects at the SEA that are related to this project? 
What is the policy or regulatory context for this project? 


What prompted this particular project? What needs does it address? Why did you decide to 
work with the Center on this? 


How did you work with the Center to develop this project? 

PROBES: 

Did your staff suggest the strategy or activities the Center might undertake to address the 
needs, or did the Center recommend the strategy? Please describe how you arrived at the 
project plan. 


Is there any part of the local context (political, fiscal, legal, economic, etc.) that you think 
played a role in the project’s early development? 

PROBES: 

How did it impact what your team was thinking and doing in relation to this project? Did 
your office have any Federal or state grant money behind this project? 


Can you tell me about the other SEA or other state agency team members on this project? 
Specifically, what are the titles or positions of SEA staff, and what are their respective roles 
on the project? 


PROBE: [Ask about other agency staff (such as workforce staff for college and career 
readiness, or human services staff for early childhood) where appropriate. ] 


Whom from the relevant Center(s) do you work with on this project? 
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19. 


20. 


Who else from outside the SEA or the Center is involved in this project? 


PROBES: 

LEAs? 

Other Centers? 

Other partners (TA providers or consultants, universities, foundations, etc.) 


How often do you meet or talk with Center representatives, or other key constituents, if 
relevant, regarding planning and implementation of the project? With which Center 
representatives do you meet? What do you discuss? How do these discussions inform 
implementation of the project? 

PROBES: 

How would you describe the roles and responsibilities of the various organizations involved 
in the project? 


Project Goals and Progress 


21. 


22. 


23. 


24. 


Ultimately, what will we “see” if this project is successful? In other words, what are your 
hoped-for outcomes from this project? 


Have the SEA needs or goals changed or shifted since the project began? [If so] Please tell 
us about that. How has this affected your work with the Center on the project? 


So far, has the project been implemented as planned? Please describe the progress of 
implementation so far, as well as any deviations from plans. What are the reasons for any 
changes in plans? 


Has any part of the local or state context changed in a way that has impacted the project’s 
implementation or success since its early development? How? 


Project Outcomes 


25. 


26. 


27. 


Have you used any products or materials produced by the Center through this project 
(including webinars, trainings, presentations)? [If so] have you been able to make use of 
what you learned? [If so] can you explain how? [If not] can you explain why? 


Do the products/materials or technical assistance events help address the project goals, and 
if so, how? 


What have been the major challenges related to this project (if any)? Please describe them. 
How were the challenges addressed (if at all)? 
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28. 


29. 


30. 


31. 


In your opinion, what has been the most successful aspect of this project? Please describe 
it. 


What have been the outcomes so far? Have you seen changes in your or your 
organization’s capacity? Please describe any changes in each of these types of capacity and 
the evidence of the change: 

e Knowledge and skills (e.g., knowledge, expertise, skills, will, understanding necessary to 
implement something) 

e Organizational (e.g., interaction, collaboration, communication among individuals in a 
system; evaluation and feedback on processes and structures for continuous 
improvement) 

e Structural/policy change (e.g., policies, procedures, and formalized practices) PROBE: 
Has your capacity to work with LEAs improved? 


Are you or the SEA as an organization doing anything differently now, that you have not 
mentioned above, as a result of this project and the TA from the Center? 


PROBES: 

How are you using new knowledge or skills? 

How has the TA influenced policies and procedures or practices at the SEA? 

How has the TA influenced the SEA’s internal interactions, collaboration and 
communications? 


Are there plans in your agency to build on, replicate, or continue this project or this line of 
work? Are you planning to do this with or without the Center(s)? What are the next steps? 


Overall, have you learned any lessons from this project's implementation? Any 
recommendations you might make to Comprehensive Centers based on either successes or 
challenges of this project? Any lessons for your own agency? 
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Outcomes-Focused Interview Protocol for Center Staff 


The OMB Control Number for this information collection is 1850-0935 and the expiration date is 04/30/2020. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average 90 
minutes response. Send comments regarding this burden estimate or any other aspect of this collection of 


information to Amy.Johnson@ed.gov . 


Introduction 


Suggested introductory remarks (Cover all main points here): 


My name is . [work for IMPAQ International, and we are conducting a National 
Evaluation of the Comprehensive Technical Assistance Centers Program. The U.S. Department 
of Education’s Institute for Education Sciences contracted with us to conduct this study. 


As part of our evaluation, we are interviewing directors, managers, and staff at the Centers to 
understand the work that you do. The purpose of the evaluation is to gather data to describe the 
work of the Centers and how the Centers build the capacities of state departments of education, 
and to report that information to the Department of Education. What you have to say is important 
to us and we appreciate your helping us understand your work. We want to assure you that 
participation in this interview is voluntary. We estimate that this interview will take 90 minutes. 


This is the last of three rounds of site visits and interviews conducted for this evaluation. 
Interviews this year will focus on the outcomes of your work with states and other constituents. 


Read the following to the respondent(s): 


“Information collected for this study comes under the confidentiality and data protection 
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 
2002, Title |, Part E, Section 183). Responses to this data collection will be used only for 
research purposes. Findings from the interview data will be reported in summary form and 
individuals will not be identified by name. Specific Centers may be identifiable when occasional 
examples of their work are provided. Other than this situation that we have made respondents 
aware of, we will not provide information that identifies you to anyone outside the study team, 
except as required by law.” 


Before we begin, do you have any questions about the purpose of the evaluation or our 
confidentiality policy? Do you mind if we audio record our session for the accuracy of our note 
taking? 


As we proceed with the interview, we will first discuss a specific project in this priority area 
(Great Teachers/Leaders or Early Learning), following up on a project we have discussed 
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previously. We will then ask a few questions about your work overall in this priority area (Great 
Teachers/Leaders or Early Learning). 


Site liaisons will have communicated when scheduling the visit that two one-hour interviews 
will be administered, one for each of the two priority areas.) 


In this interview, we are interested in focusing on outcomes. Please keep in mind that the 
purpose of this evaluation is not to monitor performance, but to describe and understand 
capacity-building processes and outcomes. 


Throughout the interview, please take care to explicitly identify the different actors and their 
different roles. We need to be able to clearly distinguish actions of the Center, SEAs, and 
partner organizations and to identify specific outcomes for specific constituents. It’s easy for 
these to blur together. 


Updated Description of Specific Project 


(Site liaisons will have communicated with the Centers before the interview to identify the 
project to be discussed. Site liaisons will prepare for the visit by reviewing the project 
information that they will be asked to confirm.) 


We would now like to discuss the project, which we have discussed in previous 
visits. 


We want to first quickly confirm the project features and understand any new developments in 
the project since our last discussion. 


1. We understand that the name of this project is . ls this the name by which the 
Center refers to and reports on the project? (If not, please provide the correct or updated 
name.) 


2. Based on our last interview in spring 2016, we understand that the goals of the project are 
, including the capacity-building goals of . ls this correct? Have there been 
any modifications to those goals or the objectives of the project since our interview in spring 
2016? If so, please describe these modifications and what led to them. Have there been any 
other important changes in the context of the project? 


3. We understand that the start date for this project was . ls this correct? What 
was the end date (or what is the anticipated end date)? 
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4. We understand that the organizational constituents for this project are (specific SEAs, LEAs, 
IHEs, other agencies or some combination, broad national constituency) 
Is this correct? If not, please provide the current organizational constituents served by the 
project. Have any of the constituents changed since our last interview in 2016? 


5. We understand that other organizations or partners involved in providing technical 
assistance on this project include . ls this correct? Have there been any changes since 
our last interview in spring 2016? 


6. The major project activities that we discussed in our last interview in spring 2016 included 
. Were there any changes in activities, or new developments in the 
implementation of the project, since our last interview in spring 2016 that we should be 
aware of? Please explain. (/f the project is not yet completed) Please describe any planned 
activities for the remainder of the project. 


Measuring Outcomes 


7. We’d like to ensure that we understand how you measure and track outcomes. 


a. How do/did you track, record, or measure changes in the SEAs’ capacity over time for 
this project? How do you assess whether the Center’s activities are helping to drive 
these changes? Do you do this differently for other project outcomes (other than 
capacity-building)? 

b. What instruments/tools, if any, do you use? 


c. What were the challenges involved in measuring outcomes for this project? How did the 
Center address these challenges? 


d. Are the measures used for this project typical of what you use as a Center for all 
projects? If there are major differences, please describe. 


Outcomes of Project 


8. We’d now like to discuss the project outcomes, and the TA strategies that produced each of 
the outcomes. 


We are now going to read through a list of types of SEA capacity and ask you to look at the 
handout for reference. Please tell us which of these categories of capacity has been 
produced by this project so far, and we will then follow up about the details. We will give 
you a chance later to describe other kinds of outcomes you achieved from this project. We 
understand that not all types of capacity are relevant for all projects. 
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(Interviewer now reads through the handout, including types of capacity and definitions. The 
respondents will have received it in advance. After the reading of the handout, the 
respondent is asked to identify the categories of outcomes produced by this project. 


For each outcome identified by the respondent, probe for a detailed description of capacity 
changes, and ensure that each of the following is addressed:) 


What is the agency or agency staff doing differently now that they did not do before? 
(Interviewer, probe for detail on specific behaviors, skills, or processes as relevant.) 


Was this an intended or unintended outcome? 


How do you know that your work led to this change? (/f the respondent has difficulty 
answering, refer to the measures discussed in question #7 and ask if they were used.) 


What technical assistance strategy(ies) or services worked well in producing this change? 
How and why? What strategies or services did not work as well? Why? 


What were the challenges in producing this outcome? How did the Center address these 
challenges? 


The handout will include these: 


a. Knowledge and skills or human capacity: Any changes in SEA staff knowledge, expertise 
or skills resulting from the project. 


o Includes increased content knowledge or policy knowledge. 
o Management or technical skills. 


b. Organizational capacity, including structural, fiscal, or material capacity: Any changes in 
SEA organizational capacity resulting from the project. By changes in organizational 
capacity, we mean any of the following: 


o SEA agency restructuring. 
o Changes in SEA agency processes and procedures. 


o Changes in communication and coordination across staff and divisions of the SEA or 
other agencies. 


o Improvements in SEA use of resources. 
o Changes in SEA performance management and continuous improvement. 
o Other. 


c. Policy development or design: Any changes in the SEAs’ capacity to develop or inform 


the development and design of state policy, including recommendations, advising to 
Boards of Education and legislators. 
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a: 


10. 


11. 


d. Capacity for policy implementation, including political capacity: Any changes in how 
SEAs implement policy. What we mean here are changes in how SEAs do any of the 
following: 

o How SEAs work to “roll out,” disseminate, or spread a policy or practice throughout 
the state. 

o How SEAs work with LEAs and other stakeholders in the state, for example, in 
supporting, guiding, or communicating with them. 

e. Other capacity changes for the SEA that you would like to describe. 


f. Capacity changes for other constituents, such as local education agencies or institutions 
of higher education. 


Has the project had any other outcomes that we haven’t discussed, including any unrelated 
to capacity building? If so, please describe. 
(Interviewer, elicit a detailed description and ensure that each of the following is addressed.) 


a. What is happening in the agency or state now that did not happen before? (/nterviewer, 
probe for detail on specific behaviors, skills, policies or processes as relevant.) 


b. Was this an intended or unintended outcome? 


c. Howdo you know that your work led to this change? (/f the respondent has difficulty 
answering, refer to the measures discussed in question #7 and ask if they were used.) 


d. What technical assistance strategy(ies) or services worked well in producing this 
change? What strategies did not work as well? Why? 


e. What were the challenges in producing this outcome? How did the Center address these 
challenges? 


(If not already addressed) Have the constituents begun to take over or own the project 
activities, information, or processes to any degree? Please explain how they are doing this 
and sustaining the work. Do you know what their next steps are for moving forward? Do 
you think they will require additional support to sustain the work, and if so, in what way? 


(For completed projects) Did the project achieve its goals? Were there any goals or project 
objectives that the Center was not able to achieve? If so, to what do you attribute this? 
(For projects still in process) \s the project meeting its goals so far? Are there any goals or 
project objectives that the Center has not been able, or may not be able, to achieve? If so, 
to what do you attribute this? 
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Priority Area Overview 


We will now ask a few questions about this overall priority area (Great Teachers/Leaders or 
Early Learning). 


12. 


Since the inception of the Center grant, has the context or focus for this priority area (Great 
Teachers/Leaders or Early Learning, as per the specific interview) changed for your 
constituents, beyond what we’ve already discussed? (If so) How has it changed? How has 
this affected the work of your Center? 


If the above project was the only one in this priority area, you may skip questions 13-15 and 
move to question 16. 


13. 


14. 


15. 


16. 


17. 


Considering all of the projects within this priority area over the last 5 years, to what extent 
and in what ways did your Center contribute to your constituents’ capacity, beyond what 
we have already discussed? What were the major outcomes that you were able to achieve 
in this area? 


What strategies were the most effective in this priority area and why? What “works” for 
building capacity in this priority area? What did not “work?” 


What challenges (if any outside of those already discussed) made it difficult to build needed 
capacity in this priority area? How did you address the challenges? Please give examples. 
How likely are these challenges (or others) to persist moving forward? What additional 
supports are needed to continue building or sustaining capacity in this area? 


Did the connections between strategies and outcomes that we have discussed today align 
with your theory of action as a Center? If not, why not? Have you changed or modified 
your theory of action (the underlying principles that inform your approach to building 
capacity) to take these results into account? Please explain. 


Have you learned anything new about how to build SEA capacity over the course of this 
grant? What lessons have you learned doing this work? 


Thank you very much for your time today. Your input has been very helpful. 
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Outcomes-Focused Interview Protocol for TA Recipients 


The OMB Control Number for this information collection is 1850-0935 and the expiration date is 04/30/2020. 


According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of 
information unless it displays a currently valid OMB control number and expiration date. Responding to this 
interview is voluntary. Public reporting burden for this collection of information is estimated to average one 
hour per response. Send comments regarding this burden estimate or any other aspect of this collection of 

information to Amy.Johnson@ed.gov. 


Introduction 


Suggested introductory remarks (Cover all main points here): 

My name is <Interviewer Name> and | work for IMPAQ International. We are researching the 
Comprehensive Technical Assistance Centers. The U.S. Department of Education’s Institute for 
Education Sciences contracted with us to conduct this study. 


(If applicable) You may recall that we talked last year about [name of Center]. 


As part of our study, we will be interviewing recipients of technical assistance from the 
Comprehensive Centers. The purpose of the study is to describe the work of the Centers, 
especially how the Centers build the capacities of state departments of education, and to help 
inform the future work of the Centers. What you have to say is important to us. We want to 
assure you that participation in this interview is voluntary. We estimate that this interview will 
take 60 minutes. 


Read the following to the respondent(s): 

“Information collected for this study comes under the confidentiality and data protection 
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 
2002, Title |, Part E, Section 183). Responses to this data collection will be used only for 
research purposes. Findings from the interview data will be reported in summary form and 
individuals will not be identified by name. However, respondents’ roles and the CC they work 
with may be identified in the report, which may lead to individuals’ being identified. Other than 
this situation that we have made respondents aware of, we will not provide information that 
identifies you to anyone outside the study team, except as required by law.” 


Before we begin, do you have any questions about the purpose of the evaluation or our 
confidentiality policy? 


To start, do you mind if we audio record our session for the accuracy of our note taking? 
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(Interviewer: Respondents will have been identified by the Centers as technical assistance 
recipients of projects in the two priority areas —teacher/leader effectiveness and early learning. 
Respondents should be staff of state agencies or offices responsible for administration of one of 
the two priority areas. In most cases the respondent will work for an office or division within the 
state department of education. Occasionally the respondent may work for a state agency other 
than the department of education, such as an office of early childhood. When administering the 
protocol, you may substitute for “state education agency” or “SEA” or “your agency or office” 
the specific name of the respondent’s office or agency as she or he generally refers to it.) 


Individual and Project Information 


1. We understand that you work for (name of agency and office or division if relevant). Is this 
correct? Please describe your position and your responsibilities. 


2. When did you first start working with the [Center name]? 


3. We'll be discussing the [project name] that you worked on with the [Center name]. We 
understand that this project (/nterviewer provide description based on recent Center 
interview.) Is this an accurate description, including an accurate description of the Center’s 
role in the project? (If not, ask the respondent to explain or add information.) If the project 
is ongoing, please describe any planned activities for the remainder of the project. 


4. Please describe your involvement in this project since it started. Were others in your agency 
involved? If so, please describe their involvement. 


5. We understand that the goals of the project for your agency are ? Is this 
correct? If not, please correct this or describe any additional goals. 


6. Have there been changes in your needs, goals or objectives for the project since its 
inception? If so, how has this affected your work with the Center on this project? 


Measuring Outcomes 


7. We’d like to understand how you measure(d) project progress and outcomes. How do you 
measure or determine whether the project is meeting its goals and objectives? Has the 
Center worked with you on measuring or tracking outcomes for the project? If so, please 
describe how they have done this. Do you need more help in measuring project progress 
and outcomes? 
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Outcomes of the Project 


8. We’d now like to discuss the project outcomes, especially the Center’s role in producing 
them. 


We are now going to read through a list of types of agency capacity and ask you to look at 
the handout for reference. Please tell us which of these categories of capacity has been 
built by the Center’s work with you on this project so far, and we will then follow up about 
the details. We will give you a chance to describe other kinds of project outcomes as well. 


Interviewer now reads through the handout, including types of capacity and definitions. After 
the reading of the handout, the respondent is asked to identify which outcomes on the list 
were produced by this project. 


For each outcome identified by the respondent, probe for a detailed description of capacity 
changes, and ensure that each of the following is addressed: 


What are you, your staff, or your agency doing differently now that you did not do 
before? (Interviewer, probe for detail on specific behaviors, skills, or processes as 
relevant.) 


Was this an intended or unintended outcome of this project (from your agency’s 
perspective)? 
How do you know that the Center’s work on this project contributed to this change? 


What evidence do you have? (/nterviewer, refer to answers under #8 if applicable.) 


What technical assistance strategies or services provided by the Center worked well in 
producing this change? How and why? What strategies or services did not work as well? 
Why? 


What are the challenges in producing this outcome (building this type of capacity change 
or other outcome in your agency)? How did the Center address these challenges? 


The handout will include these: 


Knowledge and skills or human capacity: Any changes in SEA staff knowledge, expertise 
or skills resulting from the project. 

o Includes increased content knowledge or policy knowledge. 

o Management or technical skills. 


Organizational capacity, including structural, fiscal, or material capacity: Any changes in 
SEA organizational capacity resulting from the project. By changes in organizational 
capacity, we mean any of the following: 
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e. 


o SEA agency restructuring. 

o Changes in SEA agency processes and procedures. 

o Changes in communication and coordination across staff and divisions of the SEA or 
other agencies. 

o Improvements in SEA use of resources. 

o Changes in SEA performance management and continuous improvement. 

o Other. 


Policy development or design: Any changes in the SEAs’ capacity to develop or inform 
the development and design of state policy, including recommendations, advising to 
Boards of Education and legislators. 


Capacity for policy implementation, including political capacity: Any changes in how 

SEAs implement policy. What we mean here are changes in how SEAs do any of the 

following: 

o How SEAs work to “roll out,” disseminate, or spread a policy or practice throughout 
the state. 

o How SEAs work with LEAs and other stakeholders in the state, for example, in 
supporting, guiding, or communicating with them. 


Other capacity changes for the SEA that you would like to describe. 


9. Has the project had any other outcomes we haven’t discussed, including any unrelated to 
capacity building? If so, please describe. (Interviewer, for each change described, ask the 
following:) 


What is happening in your agency or state now that did not happen before? (/nterviewer, 
probe for detail on specific behaviors, skills, policies or processes as relevant.) 


Was this an intended or unintended outcome of this project (from your agency’s 
perspective)? Did you expect the project to produce these changes, or were these changes 
unexpected? 


How do you know that the work on Center’s work on the project contributed to this 
change? What evidence do you have? (interviewer, refer to answers under #8 if applicable.) 


What activities, services or products technical assistance strategies or services provided by 
the Center worked particularly well in producing this change? How and why? Did any 
strategies or services did not work as well? Why? 


What were the challenges in working on this area of capacity? What are the challenges in 
producing this outcome (building this type of capacity change or other outcome in your 
agency)? How did the Center address these challenges? 
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10. (If not already addressed) Have you begun to take over or own the project activities, 
information, or processes, without the Centers’ help, to any degree? Please explain how 
you are doing this and sustaining the work. Do you know what your next steps are for 
moving forward? What do you think will be the biggest challenges? Do you think you will 
require additional support to sustain the work, and if so, in what way? 


11. (For completed projects) Did the project achieve its goals for your agency? Did the Center 
meet your expectations for their work on the project? Please explain. If any goals or 


objectives, or expectations for the Center, were not achieved, to what do you attribute this? 


12. (For projects still in process) \s the project meeting its goals so far? Is the Center meeting 
your expectations for their work on the project so far? If any goals or objectives (or 
expectations for the Center) have not been achieved, to what do you attribute this? 


Priority Area Overview 


We’d now like to ask about your work with the Center in the area of /teacher/leader 
effectiveness or early learning] overall. 


13. Since 2012, has the local or state context or focus for this priority area (teacher/leader 
effectiveness or early learning, as per the specific interview) changed for your agency? If 
yes, how has it changed? How has this affected your work with the Center? 


14. Do you believe the Centers have been effective in addressing the needs of the SEA and the 
state in this priority area? If yes, please describe how and what evidence you have of their 
effectiveness. If no, please explain why and how/in what ways they have not been effective 
in serving the SEA/state needs. 


15. Considering all of your work with the Center within this priority area over the last 
[applicable number] years, to what extent and in what ways did the Center contribute to 
your agency’s capacity? What were the major outcomes that achieved in this area? 


16. What Center technical assistance strategies, services, or products were the most effective in 


this priority area and why? What “worked” for building your agency’s capacity in this 
priority area? What did not work? 


17. What challenges (outside of those already discussed) made it difficult to build your agency’s 


needed capacity in this priority area? Please give examples. How likely are these challenges 
(or others) to persist moving forward? What additional supports might you need to 
continue building or sustaining capacity in this area? 
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18. Do you have any lessons or recommendations for the Comprehensive Centers based on 
your experience in this project or the priority area overall? Did you learn any lessons for 
your own agency? Please explain. 


Thank you very much for your time today. Your input has been very helpful. 
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Interview Codebook -— All Interviews 


The coders applied these codes to interview transcripts, or parts of transcripts, and monitored intercoder reliability using methods 
described in Appendix B. 


The coders were instructed to read through each interview transcript, coding paragraphs or passages of text to a// applicable codes 
or subcodes. It was permissible to assign more than one code to the same selection of text. 


Interview Protocol 


py=yiTayi ale} 


Design 


Category 
Capacity- 
Building 
Principles 


How the Center conceptualizes capacity 
building—including the meaning of capacity 
building, how the Center works with the SEA 
to build capacity, and the themes of TA that 
are specific to capacity building and are 
different from other types of TA. 


Example/Quotation from Data 


Fostering Long- 
Term Changes 


Fostering 
Ownership 


Center focuses on long-term changes in the 
SEA or changes that are embedded in SEA 
systems and not easily dislodged. This code 
can be applied to capacity building that goes 
beyond near-term, quick-turnaround 
changes or to capacity building that is both 
long-term and systemic (and therefore 
sustainable). 

Center focuses on enabling SEA staff to take 
over initiatives and ultimately run them 
without the Center’s help. 


The Center helps the SEA change in ways 
that are long lasting despite staff turnover. 
Many Center staff mentioned that capacity 
building means long-term change in a large 
system, such as changes in state standards, 
assessments, teacher evaluation policy, or 
relationships between SEAs and LEAs. 


“TSEA staff] phase us out.” “When we see it 
happen, really, they start to take over the 
mechanisms of the project itself. We call it a 
gradual release model ... It’s a much slower 
process. They really have to understand and 
take hold and get an idea. They’re part of 
the building of the project, but they’re 
usually not ready to do any leading of it, 
taking charge of things. That’s what they 
term it too.” 


Fostering 
organizational 
process change 


Center work creates changes in fundamental 
systems or practices; it produces broadly 
applicable knowledge, processes, 
frameworks, or tools. SEAs or LEAs gain 


One respondent described how the Center 
trained SEA staff by leading a process for 
grant review that the SEA could use going 
forward. Another Center developed a 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


something that they can apply to a variety of 
situations or processes going forward, rather 
than just filling a short-term need. Centers 
may provide a framework that includes both 
a set of considerations or principles and a 
process for making decision May also be 
defined as “institutionalizing” processes or 
practices. 


framework that SEAs can use when 
implementing new standards and 
assessments. Another respondent: 
“Essentially, our definition of capacity 
building is increasing, in our case, a SEA’s 
ability to successfully undertake an initiative 
and then to learn from that process and 
apply those lessons to future initiatives.” 


Design, 
Implementation, 
Outcomes 


Capacity 
Types/Dimensio 
ns 


Types/levels of skills and practice, behavior, 
or policy changes the Center is trying to 
achieve. If respondents give examples of 
outcomes to illustrate the capacity-building 
design, those can be included here. (Don't 
confuse this code with code 1, Capacity- 
Building Conceptual Definition, which covers 
the broad concepts.) 


See below. 


SEA Staff 
Individual 
Knowledge and 
Skills 


Increase knowledge of policy content and 
options, exemplars, and lessons from other 
states; increase technical skills; increase 
management and leadership skills. 


SEA Organizational 
Capacity/ 
Development 


Primarily internal changes in agency or 
cross-agency structures, processes and 
practices. (Don't include work with LEAs--- 
this goes under policy implementation.) 


See types of organizational capacity below 


Communication 
and Coordination 


Improved coordination with SEA or 
between SEA and other agencies 


SEA is making progress in “breaking down 
silos” to work better across divisions or 
agencies. 


Agency Redesigned/expanded capacities of SEA Capacity to create and manage a new 
restructuring divisions and structures division, such as an early learning division. 
Continuous Capacity for continuous improvement of Creating processes of self-evaluation; 
improvement agency creating performance metrics. 

Policy Design SEA is better able to develop, design, SEA develops capacity to design ECE teacher 


change, interpret policy or regulations 


licensing/permit policy, with Center support 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


Policy 
implementation 


SEA increases capacity to implement policy 
in the state, especially through guidance and 
training of LEAs and schools 


SEA is able to train districts on hiring highly 
effective teachers 


Design 


Needs 
Assessment 


How the Center assesses SEA needs for 
Center TA, including assessment of existing 
SEA capacity to meet its goals. 


See level 2 and 3 codes. 


Data Review 


Meetings with SEA 


Center reviews existing data (such as 
student performance data) or studies that 
identify states’ educational needs. 


In-person or telephone meetings with SEA 
leaders and/or staff. 


Center may reference data in a particular 
report such as a state inspector general’s 
report or a report on national or state 
standardized test scores. 

“Facilitated meetings twice per year plus 
weekly or monthly phone calls with project 
staff.” 


Surveys 


Surveys of constituents to assess needs. 


Annual surveys or questionnaires 
administered before and after specific 
events. 


Discussions with 
State Chiefs 


In-person or telephone meetings with chief 
state school officers (commissioners, 
superintendents, depending on the state). 


May be individual meetings/calls, group 
retreats, or both. 


Relationship 
Building 


Center staff develop relationships with 
chiefs and SEA staff (and others) and, 
through ongoing relationships, learn about 
and understand needs. 


Content Expertise 


Center staff (especially Content Centers) 
work in the priority area (e.g., early 
learning), understand national priorities and 
policies, and know the issues/ needs that 
states are facing. 


“It is our business to know the emerging 
needs in the field.” Some respondents 
referenced their knowledge of the most 
recent national studies. 


Tracking Requests 


Centers have a system for monitoring and 
recording the requests that they receive 
from SEAs. 


Several Centers have online tracking systems 
accessible to all staff. They may see “a 
pattern of needs” emerging that they can 
address. 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


Work with Partner 
Organization 


SEA Initiated 


Centers draw on specialized organizations 
with intimate knowledge of state or content- 
specific needs. 


SEA initiates contact or topic with the 
Center, identifying a particular need for 
help. 


Content Centers rely on Regional Centers to 
assess regional and state needs; Regional 
Centers rely on Content Centers to assess 
content-specific needs and share expertise. 
“They [the SEA] now contacts us when they 
have a TA need.” 


Center Staff Staff of a few Centers were located in SEA Both Centers and TA recipients have 
Embedded with offices in order to conduct ongoing needs described a few of these situations where 
SEA sensing Center staff work “on-site” at the SEA. 
Design Developing Developing goals and strategies for each 
Center Work state 
Plan/Choosing 
Strategies 
Focus, efficiency Consideration of Center expertise, budget, “We go deep rather than wide”; “We think 
staff, time; targeting to highest priorities about priority in terms of resources and in 
and greatest likelihood of outcomes terms of our ability to build to capacity ...” 
Interactive Back-and-forth between Center and SEA “A continuous dialogue assessing Center 
Planning Process staff or state leaders informs TA plans. strengths and state needs and where’s the 
best fit.” Some Centers use the term 
“transactional” to describe the process. 
Build on Center understands the SEA or constituent if you come in and try to ... sell an initiative 
Relationships and well enough to match strategies to that doesn't align with their [situation], that 
State/Local needs/capabilities/goals. will likely not go anywhere 
knowledge 
Design, Broad TA Broad strategies for building capacity 
Implementation, Strategies 
Outcomes 
Cross-Policy The Center helps the SEA understand how Center supports the application of 


Coordination 


policy areas intersect, helps the SEA 
coordinate with other agencies, or helps SEA 
divisions work together. 


curriculum standards to early childhood. 


Modeling 


Modeling of a process by the Center in order 
for the SEA to learn and adopt a new 
process it can implement on its own. 


Center organizes professional development 
presentations/work sessions for the SEA, 
using a format that the SEA can then 
replicate for use with LEAs. 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


Stakeholder Includes, but not limited to, meeting 
Engagement/ facilitation. Center helps SEA conduct 
Facilitation outreach, convene meetings, obtain input. 
Cross-State May include a variety of Center-run regional | Center supports peer learning networks. 
Knowledge activities, sharing or dissemination of state SEAs learn promising school turnaround 
Building promising practices. Includes Center strategies from each other. 

activities such as training, web scans, 

resource libraries that synthesize 

information from various states. SEAs may 

or may not be actively meeting or working 

together. 
Thought Center acts as a "critical friend" and provides | Center staffer: "I think we've also helped 
Partnering a different perspective for SEA, helps SEA them bring together a space where we can 


with problem solving and brainstorming; 
SEA intentionally brings Center in to tackle a 
particular issue. 


exchange ideas and brainstorm about what 
is best, and bring in resources from the 
outside ... helping to give them another 
perspective." 


Implementation 


TA Activities 


Activities, products and services that 
operationalized the strategies 


Products, tools, 


Center-provided products may include policy 


Center produces publication summarizing 


resources scans, policy briefs, guidance documents, research on best practices in teacher 
reports, toolkits evaluation. 
Webinars Web-based meetings or trainings Web-based training on teacher data use 


In-person training 
or conference 


Center organizes, delivers in-person sessions 


Support for Help SEA conduct, use research or arrange Help with survey design and administration, 

research for experts to conduct research help with analysis, manage pilot study of 
teacher evaluation 

Meeting Overlaps with broad strategy on stakeholder 

facilitation management 


Implementation 


Collaboration 


Collaborating with other organizations on 
projects, especially other Centers 


Cross-Center Type 


Regional and Content Centers work together 


The Centers co-facilitate project, with 
Regional Center providing state 
knowledge/access and Content Center 
providing content knowledge 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


Same Center Type 


Regional Center with another Regional 
Center, or Content Center with another 
Center 


Collaboration enables Centers to extend 
their geographic reach or combine content 
areas 


Outcomes Outcome Measures used to track project outcomes. 
measures Include any evidence or data that supports 
claim of outcomes 
Surveys Constituent Surveys include capacity- Center administers surveys to SEA staff 
building (C-B) outcome measures members before and after project that 
measure knowledge, skills, attitudes, 
behaviors on a particular topic 
Informal Methods | Ad hoc discussions with constituents, Center staff say "in our conversations with 
anecdotes, informal observations states, they tell us that the conference was 
very helpful" 
Interviews Structured interviews Center does structured interview, following 
a protocol, with SEA staff about their 
knowledge of pre-K assessments 
Case Studies Structured case studies Center staff systematically collect data from 
SEAs to examine changes in CB - includes 
interviews with key staff and collection of 
documents and surveys 
Capacity -Building | Application of structured capacity-building Center has a capacity inventory rubric with a 
Rubric or matrix scale with indicators and performance levels | scale: 1 = little or no development and 
implementation to 4 = exemplary level of 
development and implementation 
Outcomes Goal To what extent did project meet its goals? "There really wasn’t anything we really 


Achievement 


(or is meeting its goals, if still in process) 


implemented that didn’t come to fruition, so 
| think we pretty much met all of our goals." 


Implementation, 
Outcomes 


Challenges 


Key Challenges to achieving specific 
outcomes. Only apply if TA recipient 
specifically identifies the challenge. Do not 
over interpret. If you cannot find a child 
node that fits, code to parent node & 
"Other" and annotate. 


SEA turnover 


Staff turnover at SEA (or TA Recipient) 
organization 


New SEA person coming into role and having 
to re-orient to the project work, and get to 
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Interview Protocol 


Category 


Definition 


Example/Quotation from Data 


know the Center and build relationship with 
the Center staff 


SEA 
staff/time/resourc 
e constraints 
Changing state 
policy or priorities 


Difficulty scheduling time for SEA staff to 
participate 


Shifts in state priorities/strategies that 
disrupt project 


Constrained timeline, busy schedules, 
difficulty finding time for all to be involved 
and participate in project activities 

"The work that we did with the XX 
Commission was under our previous 
superintendent, and with the transition of 
the administration, they sort of, you know, 
wanted to wipe the slate clean and so they 
even removed the report from our web site 


Center resources 


Includes financial or organizational 
resources; Center lacks sufficient resources 
to carry out the project as fully as desired 


State had to bring in another TA provider 
because Center did not have budget/staff to 
continue or provide service any longer. 


Misunderstanding 
of Center role 


Breakdown in communication or 
understanding; SEA may not understand the 
Center's scope; Center finds it difficult to 
establish relationship with SEA 


Center has difficulty getting SEA to respond 
or to work together on a state plan because 
SEA staff are not familiar with Center/do not 
know how to use the Center. 


Mismatch 
between SEA & 
Fed priorities 


Federal policy, regulation or priorities 
change that affect the project 
progress/implementation 


ESSA caused major shift in SEA's work 


Implementation, 
Outcomes 


Address 
Challenges 


How Center addresses challenge. Double 
code with specific challenges when possible 


Persistence 


Center staff follow up consistently with 
constituents 


Center staff maintain frequent, flexible 
communication, follow up on a regular basis 
and engage in creative problem solving with 
their SEA contacts to help keep attention on 
the project despite distractions or shifts. 


Stakeholder Center builds on relationships with multiple | Center builds relationships with mid-level 
Relationships/ stakeholders (including SEA and others) in SEA staff and community organizations as 
Broad Buy-In order to foster long-term investment well as with top leadership; also helps SEA 
despite SEA turnover. build support with multiple constituencies 
Flexibility Center makes adjustments in response to Center shifts project focus when state 


shifting priorities 


funding for original initiative is cut off. 
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Interview Protocol 


Definition 


Outcomes 


Outcomes 


Category 
Effective/ 
Ineffective 
Strategies 
Lessons Learned 


See TA Strategies and Activities Above. 
Double code with “Effective” or 
“Ineffective” 

Lessons learned or reinforced about capacity 
building, includes recommendations for the 
Centers 


Example/Quotation from Data 


Important to 
engage multiple 
stakeholders 


Successful TA requires involvement of diverse 
stakeholders within and/or beyond the SEA 


Respondents report importance of going 
beyond the immediate constituent to 
engage (for example) multiple divisions 
within SEA, multiple agencies and 
organizations statewide, or district staff, 
teachers, and parents 


Need for guidance 
on use of Centers 


TA recipients had initial difficulty identifying 
role for Centers and/or respondent 
recommends more upfront guidance or 
information about Centers 


Respondents may refer to the confusing TA 
landscape and the need for more structured 
introduction to Centers (such as “menu of 
options”) 


Increase/clarify 
focus on LEAs 


TA should help the SEAs work with LEAs and 
ultimately produce outcomes at local level 


Respondents suggest TA needs to focus on 
SEA-LEA relationship, clarify what SEA aims 
to produce at LEA level 


Promote Aim of Center TA should be to create May mention need for Center to equip the 

sustainable knowledge and practices that constituent can | constituents with tools, documents, 

practices apply to future initiatives processes that will outlast staff changes and 
priority shifts 

Need for Regardless of capacity improvements, SEAs Respondents note SEAs are not ready, or are 


continuing support 


seek continuing Center support 


too small, to carry the initiatives forward 
entirely without Center help 
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Codebook for Technical Assistance Activity Types 


The categorization scheme for the type of TA activity was determined by reviewing several 
fields from the first round of activity reports, including the activity description, activity 
milestone, service or product developed, project name, priority area, project goals, and TA 
type. Evaluators read and analyzed the data to look for distinct categories of TA types; they 
then developed definitions of the TA types that emerged. After the first data collection cycle, 
evaluators created drop-down menus for these categories so that they could be selected by 
Center survey respondents in subsequent rounds of data collection. 


TA Type 


(Activity Level) 
Conference/ 
Summit 


Definition 

Engaging in activities leading to a convening 
held or organized by the Center; presentation at 
a conference 


Examples 

Center cosponsored equity leadership 
conference with outside partner 
organization. 


Consultation 


Serving as thought partner or content expert; 
reviewing resources and giving feedback 
(providing examples or information but not 
developing materials or deliverables); holding 
discussions or initial conversations; providing 
direct counsel or advice to stakeholders and 
partners; fielding calls 


Center advised SEA in determining which 
early learning initiative logic model 
outcomes are appropriate to measure and 
the processes and structures for 
measuring those outcomes. 


Center gave SEA examples of peer 
learning communities. 


Brokering 
Stakeholder 
Connections 


Arranging connections between stakeholders, 
such as between SEAs, or between SEAs and a 
Content Center, REL, or other organization, 
mainly independent of meetings 


Center secured experts to provide TA to 
SEA as requested. 

Center introduced SEA to director of preK 
intermediary organization. 


Data Analysis 


Analyses of data from interviews, focus groups, 
surveys, etc.; leading root-cause analyses; 
summarizing findings 


Center shared summary of teacher survey 
results on English learning effectiveness 
and evaluation. 


Literature 
Review/ 
Reference List 


Reviewing research or documents and preparing 
summaries leading to producing a tangible 
resource; conducting scans; developing collections 
of resources 


Center developed annotated bibliography 
on overview of approaches to 
kindergarten entry/readiness 
assessments. 


Meeting Taking a specific role in meetings, such as Center facilitated the early learning 
Facilitation leading, organizing, or coordinating meetings advisory committee’s executive team 
with external stakeholders to help groups define | quarterly meetings. 
and achieve meeting objectives. These events 
often convene multiple stakeholders with 
coordination by the Center. Planning meetings 
and preparing agendas. 
Resource Generating new written resources designed to Center produced first chapter of multiple- 
Provision support stakeholders, including reports, chapter series and created a product that 
presentations, and others will walk SEAs through a process to select 
their measurement goals and measures. 
Survey Designing and implementing surveys Center generated ideas and examples of 


alternative ways to design a principal 
survey and shared these ideas with SEA. 
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TA Type 
(Activity Level) 


Definition 


Examples 


Webinar Hosting, leading, developing, or participating in | Center facilitated pre-planning meeting 
a webinar (the term “webinar” will be used) for a webinar on principal evaluation 
systems. 
Workshop/ Leading or implementing a workshop or Center held SEA Race to the Top 
Training training; conducting staff orientations; facilitators’ training. 
conducting professional learning; coaching; 
leading in-person work sessions 
Web Completing wireframes for website; reviewing Center provided technical support to SEA 
Development or developing websites, including SharePoint regarding transferring paper-based system 
Support sites; developing online media resources to online system. 
Other This category captures all events that were not | Center edited an SEA’s video professional 


captured in the categories above. 


development module. 
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APPENDIX D: Additional Analyses 
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Exhibits D-1 through D-25 provide additional details supporting or expanding on the findings 
presented in the report. 


Design 


Exhibits D-1 through D-3 present results of design-focused interviews conducted with staff of 
the 22 Centers in 2015. Qualitative interview data were coded using the methods described in 
Appendix B. Code definitions are provided in Appendix C. 


As illustrated in Exhibit D-1, all Centers reported that SEA staff knowledge and skills were a 
target of capacity building. Most Centers identified SEA organizational capacity and SEA 
capacity for policy implementation as additional dimensions of capacity that they targeted. A 
smaller number of Centers (eight) identified SEA capacity for policy design. 


Exhibit D-1. Centers’ Capacity Building Definitions: Dimensions of Capacity 


Centers’ Targeted Dimensions of SEA Capacity 


SEA Staff's SEA SEA Capacity | SEA Capacity for 
Knowledge _ | Organizational for Policy Policy 
and Skills Development Design Implementation 


Regional Center 1 


Regional Center Total 15 


Content Center Total 
Source: Center design-focused interviews, 2015. 
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Exhibit D-2 shows that most Centers also defined capacity building in terms of one or more of 
the principles of fostering ownership, promoting longer-term change, and promoting change in 
organizational processes. One Content Center did not identify these principles but focused its 
definition on the dimensions of capacity presented in Exhibit D-1, as they applied to the specific 
content area of this Center. 


Exhibit D-2. Centers’ Capacity Building Definitions: Principles of Capacity Building 


Centers’ Principles of Capacity Building 
Fostering Fostering 
Longer-Term | Organizational 
Changes Process Change 


Fostering 
Ownership 


Regional Center 1 
Regional Center 2 
Regional Center 3 
Regional Center 4 
Regional Center 5 
Regional Center 6 
Regional Center 7 
Regional Center 8 
Regional Center 9 
Regional Center 10 
Regional Center 11 
Regional Center 12 
Regional Center 13 
Regional Center 14 
Regional Center 15 
Regional Center Total 


me IO1OaIODIONTa lala 
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mae |S [Ss /S 15/5 |S [= 
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1s 2 12/3 ]2 /2 {E 
me ae ae ee ee ee 
=| |o |o |o |o |o ]o |o 
=> a a ee a en ee 
Oe a ae atl bel el La 
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Source: Center design-focused interviews, 2015. 
Exhibit D-3 illustrates needs assessment methods that Centers reported using. All Centers, with 


one exception, reported meetings with SEA staff and/or chief state school officers as a needs 
assessment method, but most Centers also supplemented with additional methods. 
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Exhibit D-3. How Centers Assessed SEA Needs 


Meetings Center's 


State- with Chief | Knowledge Center Staff 
Meetings | Initiated | State School | of Needs in Data Tracking Embedded 
with SEAs | Request Officers Field Review /|Partnerships| Requests | Surveys w/SEA 


Regional Center 1 
Regional Center 2 


Regional Center 3 
Regional Center 4 


Regional Center 5 
Regional Center 6 
Regional Center 7 
Regional Center 8 


Regional Center9 | | 
Regional Center10_ | | 


Regional Center 11 
Regional Center 12 
Regional Center 13 
Regional Center 14 
Regional Center Total 12 4 
Content Center 1 
Content Center 2 
Content Center 3 


Content Center Total 6 5 5 


Total 18 14 14 


Source: Center design-focused interviews, 2015. 


As stated in the report, almost all Center projects served SEAs, but most served other TA 
recipients as well, and projects serving LEAs increased over time (see Exhibit D- 4). 


Exhibit D-4. Affiliations of TA Recipients of Center Projects 


nee 2015 2016 2017 
Recipient Type : : ; 
N= 152 projects N=187 projects N= 186 projects 
SEA 


93% 94% 95% 
LEA 16% 27% 43% 


Regional Center | 26% 16% 21% 
Non-SEA state agency — 8% 12% 6% 
Content Center 3% 6% 5% 
Other 28% 28% | 29% 


Source: Center activity reports, 2015-2017. 

Percentages of projects serving these TA recipients in any project activities. Percentages can add up to more than 
100% because projects could serve more than one constituent group. “Other” recipients include institutions of 
higher education, research centers, and professional associations. 
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Implementation 


Implementation data were collected through interviews, activity data, and Center staff survey 
data. Interview findings on Centers’ broad TA strategies that were used in a sample of projects 
identified by the Centers to discuss are presented in Exhibit D-4. Activity reports provided more 
granular data on TA activities of 333 projects implemented across three years (Exhibits D-5 and 
D-6). The Center staff survey responses on collaborations with non-Center organizations 
(supplementing the cross-Center collaborations reported in the report) are presented in Exhibit 
D-7. 


TA Strategies and Activities 

When describing TA strategies of projects that were the focus of interviews, Center staff most 
often identified thought partnering, followed by cross-state knowledge building, and 
stakeholder engagement; TA recipients most often identified stakeholder engagement, 
followed by thought partnering and cross-state knowledge building (Exhibit D-5). TA strategy 
code definitions are presented in Appendix C and are also discussed in the report. 


Exhibit D-5. Implementation of Broad TA Strategies: Interview Results 


Percentage of projects for which strategy was identified 
(N = 44 projects discussed by both Center staff and TA recipients) 


TA Strategy Center Staff Interviews TA Recipient Interviews 
Thought partnering 82% 68% 
Cross-state knowledge building 75% 68% 
Cross-policy coordination 52% 23% 
Stakeholder Engagement/Facilitation 73% 82% 
Modeling 48% 23% 


Source: Center staff and TA recipient implementation interviews, 2015 and 2016. 
Qualitative data from interviews were coded using methods described in Appendix B. TA strategy codes are 
described in Appendix C. 


Exhibit D-6 presents the frequencies of TA activity types documented in activity reports, 
supporting the findings presented in the report. As noted in the report, projects most often 
included consultations and meeting facilitation activities. Other activities that were relatively 
common included product and resource provision, workshops, and brokering of stakeholder 
connections, which involved outreach, connections to experts and peers in other states, and 
arranging partnerships. 
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Exhibit D-6. Types of TA Activities in Center Projects 


Percentage of Projects Including Each Activity 


Technical Assistance Activities All Centers Regional Centers Content Centers 
(N = 333) (N = 188) (N = 145) 
Consultation 68% 78% 55% 


Meeting facilitation 50% 59% 39% 
Product/resource provision 34% 
Workshop/training 24% 


Brokering stakeholder connections 17% 
Conference/summit 16% 
Webinar 17% 
Data analysis 8% 
Web development support 18% 
Survey 4% 
Literature review/reference list 8% 


Source: Center Activity reports on projects in the early learning and teacher/leader effectiveness priority areas, 
2015-2017. 
TA activity codes are defined in Appendix C. 


As presented in Exhibit D-7, most projects included at least two different types of TA activities, 


and Regional Centers were more likely to include three or more activity types than Content 
Centers. 
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Exhibit D-7. Mix of TA Activity Types per Project 


Percentages of Projects 


Number of Activity Types All Centers Regional Centers Content Centers 
(N = 333) (N = 188) (N = 145) 


One Activity Type 25% 19% 32% 
Two Activity Types 25% 20% 31% 

Three or More Activity Types 50% 37% 
Source: Center Activity reports on projects in the early learning and teacher/leader effectiveness priority areas, 
2015-2017. 


Exhibit D-8 presents the distribution of the specific content focus, within the two key priority 
areas, of Center TA projects included in the activity reports. In the early learning area, projects 
most often focused on early learning quality improvement. In the teacher/leader effectiveness 
area, projects most often focused on educator equity plans. 


Exhibit D-8. Content Focus of Projects 


Teachers/Leader Effectiveness (N = 236 Projects) 

Equity plans and equitable access to quality 20% 28% 
teachers 

Principal/leader evaluation 1% 1% 
Principal/leader preparation/pipeline/recruitment 3% 5% 
Principal/leader professional development 11% 15% 
Principal/leader standards 2% 2% 
Teacher evaluation 13% 18% 
Teacher preparation, pipeline, and recruitment 3% 4% 
Teacher professional development 16% 22% 
Teacher standards 2% 3% 
Other 2% 3% 
Early Learning (N = 97 Projects) 

Early childhood education quality improvement 12% 41% 
Early childhood education workforce 7% 24% 
Early childhood assessment and standards 6% 22% 
Early childhood data systems or data use 3% 9% 
Other 1% 4% 


Source: Activity reports, 2015-2017. 
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Collaborations 

Exhibit D-9 presents Center staff survey reports on project collaborations with non-Center 
organizations. These collaborations supplemented collaborations between Centers, as 
discussed in the report. Many projects reported several types of collaborations. 


Exhibit D-9. Non-Center Organizations Collaborating on Center Projects 


Percentage of Projects 


All Centers Regional Centers Content Centers 
(N= 279) (N= 171) (N= 108) 


Types of Organizations 


Non-Center non-REL TA provider 29% 30% 27% 
REL 22% 27% 13% 
Institution of higher education 20% 21% 19% 
Other 30% 29% 31% 


Source: Center staff survey, 2015-2017. 

Percentage of projects for which at least one Center staff reported collaboration with this type of organization. 
REL refers to Regional Educational Laboratories. “Other” includes professional associations, nonprofit 
organizations, and research centers. 


Challenges 
Center staff identified a variety of challenges in addition to the top challenges included in the 
report. Exhibit D-10 shows complete Center staff survey results for project challenges. 
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Exhibit D-10. All Project Challenges Reported by Center Staff 


Percentage of Projects 
Regional Content 
Centers Centers 
(N= 279) (N= 171) (N = 108) 
Changing priorities at an SEA 54% 56% 52% | 
Turnover among state education agency (SEA) staff 52% 55% 48% 
Meeting diverse needs across SEAs (and/or regions) 43% 40% 47% 
Cuts in SEA staffing/budgets 42% 41% 44% 
Project timeline constraints 39% 49% 24% 
Policy shifts in a state government 38% 36% 40% 
Turnover in chief state school officer and/or other SEA leadership 35% 37% 31% 
Lack of communication within an SEA / 34% 33% 34% 
Lack of Center financial resources / — 24% — 21% : 28% 
17% 16% | 19% 


Project Challenges Reported by Center Staff All Centers 


Constituent committment or engagement | 11%) 12% 10% 
Mismatch between stated SEA needs and Center federal priorities 10% 8% 15% 
Lack of access to appropriate expertise i 10% 9% | 12% 
Turnover among Center staff | 9% 11% 5% 
Competing priorities | 8% 8% 7% 
Data/information access 1 4% _ 4% 4% 
Other 13%. 9% 19% 


Source: Center staff surveys, 2015-2017. 

Project challenges reported by at least one staff member for each project. 

“Other” challenges included limited SEA staff in the relevant policy area, delays in receiving information from SEA 
staff, political tensions, lack of communication across state agencies or between state and local agencies. 


TA recipients were asked to select “all that apply” from a list of challenges they might have 
encountered in working with the Center. Exhibit D-11 shows that results were similar across years. 
Difficulty scheduling time to participate, staff turnover (in SEAs), and unclear understanding of the 
Centers’ role were the most frequently identified challenges throughout. However, in each year, 
over 60 percent of respondents did not report any challenges. 
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Exhibit D-11. Challenges Reported by TA Recipients, by Year 


Percentage of TA Recipients Reporting Challenge 
All Centers Regional Centers Content Centers 

ors e 

Difficulty scheduling time for our staff to participate 12% 13% 10% 

Staff turnover at our organization 11% 14% 5% 

Unclear understanding of role of the Center 9% 9% 9% 

Mis-alignment between our needs and Center priorities | 7% 5% 9% 


Challenges 


Insufficient number of in-person meetings with Center 5% 2% 10% 
Delay in receiving needed TA 3% 4% 2% 
Lack of Center understanding of our needs 3% 5% 1% 
Staff turnover at Center 2% 2% 2% 
Lack of skills/expertise by Center staff 2% 3% 0% 
Difficulty in communication with Center 1% 1% 2% 
Total selecting at least one challenge 37% 38% 
ee a =v = 214 cae 

Difficulty scheduling time for our staff to participate 10% 16% 

Staff turnover at our organization 12% 17% 

Unclear understanding of role of the Center 7% 

Mis-alignment between our needs and Center priorities | 6% 

Insufficient number of in-person meetings with Center 3% 

Delay in receiving needed TA 2% 

Lack of Center understanding of our needs 4% 

Staff turnover at Center 2% 

Lack of skills/expertise by Center staff 3% 

Difficulty in communication with Center 2% 

Total selecting at least one challenge 29% 

N =231 

Difficulty scheduling time for our staff to participate 14% 

Staff turnover at our organization 9% 

Unclear understanding of role of the Center 8% 

Mis-alignment between our needs and Center priorities | 5% 

Insufficient number of in-person meetings with Center 4% 

Delay in receiving needed TA 4% 

Lack of Center understanding of our needs 4% 

Staff turnover at Center 2% 

Lack of skills/expertise by Center staff 2% 

Difficulty in communication with Center 3% 

Total selecting at least one challenge 


Source: TA recipient surveys, 2015-2017. 
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Outcomes 


TA Recipient Ratings 

Outcomes findings were based on survey data and interview data. While Center staff survey 
data were aggregated at the project level across years (see the report for outcomes reported by 
Center staff), TA recipient survey data were analyzed at the respondent level, separately for 
each of the three survey years. Many of the exhibits below add supporting detail to TA recipient 
survey findings in the report, presenting TA recipient survey results by year. 


TA recipients’ overall ratings for each capacity type varied little across years (Exhibit D-12). 


Exhibit D-12. TA Recipients’ Ratings of the Extent to which Centers Contributed to Specific 
Types of Capacity, All Centers, By Year 


Between toa Between to some ) Between none and 
moderate and to extent and toa to some extent (0-1) 
a great extent moderate extent 

(2.1-3) (1.1-2) 


Knowledge and rr el 
Skills GE | 
ES 1 | 


Organizational 
Capacity 


Capacity for 
Policy Design 


Capacity for 
Policy 
Implementation 


Source: TA Recipient Survey, 2015-2017. 
Ratings of contributions to each capacity type were based on average ratings of items associated with that type 
(see Exhibits D-13 through D-17). 


Exhibit D-13 presents results by Center type by year, separately for contributions to individual 
knowledge and skills, organizational capacity, capacity for policy design, and capacity for policy 
implementation. Over the three survey years, ratings of contributions to individual knowledge 
and skills varied little for TA recipients of either Center type. Ratings of contributions to 
organizational, policy design, and policy implementation domains improved over time among 
TA recipients of Content Centers. 
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Exhibit D-13. TA Recipients’ Ratings of the Extent to which Centers Contributed to Specific Types 
of Capacity, by Center Type, by Year 


Knowledge and Skills 


Regional Centers 


2015 2016 2017 
Score Range 
a 129) (N= _ (N=113) 


Paversgs sss 


Content Centers 


2015 2016 2017 
Score Range 
(N=78) (N=98) (N=114) 


A 
Average SS ey ey Sal Pe re 
Organizational Capacity 


Regional Centers 


2015 2016 2017 
Score Range 
a 127) (N=110) (N=110) 


a a 


Content Centers 


2015 2016 2017 
Score Range 
(N=73) (N=94) (N=111) 


[average a9 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program D-12 


Policy Design 
Regional Centers 


Content Centers 


Policy Implementation 
Regional Centers 


Content Centers 


Source: TA recipient survey. 

Based on the average of all items in the individual questions listed in Exhibits D-14 to D-17 for each type of 
capacity. TA recipients were asked to rate the Centers’ capacity-building for each item using the following four- 
point scale: not at all (0), to some extent (1), to a moderate extent (2), to a great extent (3). 


Exhibits D-14 through D-17 present the TA recipient survey items that were used to measure 
each of the four types of capacity. Item-level ratings from 0 (Centers did not contribute) to 3 
(Centers contributed to a great extent) were averaged to produce results for each capacity type 
(Exhibit D-12). The exhibits below present disaggregated item-level results by Center type and 
by year, presenting the proportion of respondents who gave the highest ratings. 
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Exhibit D-14. TA Recipients’ Item-Level Ratings of Centers’ Contributions by Year: 
Knowledge and Skills 


To what extent do you agree that the technical assistance you Percentage of Respondents Reporting 


received during the last 12 months helped you in the following “To a Moderate Extent” 
ways? OR “To a Great Extent” 


Content 
Centers 


Regional 
Centers | Centers 


Solve a problem 
Access the information you wanted 

Increase your knowledge and expertise in the topic 

Identify additional training events, webinars, and seminars of interest 
Build collaborations with professional experts outside your organization 
Access useful information or ideas from other states 

Learn about or obtain new tools that you can use in your work 


Learn new strategies that you can use in your work 


Perform your job in a more informed and efficient way 


All Regional | Content 
Centers | Centers | Centers 


Solve a problem 
Access the information you wanted 

Increase your knowledge and expertise in the topic 

Identify additional training events, webinars, and seminars of interest 
Build collaborations with professional experts outside your organization 
Access useful information or ideas from other states 

Learn about or obtain new tools that you can use in your work 

Learn new strategies that you can use in your work 


Perform your job in a more informed and efficient way 


2017 


Solve a problem 


All Regional | Content 
Access the information you wanted 
Increase your knowledge and expertise in the topic 
Identify additional training events, webinars, and seminars of interest 
Build collaborations with professional experts outside your organization 
Access useful information or ideas from other states 


Learn about or obtain new tools that you can use in your work 


Learn new strategies that you can use in your work 


Perform your job in a more informed and efficient way 
Source: TA recipient survey. 

Total N = Number of respondents selecting a rating; respondents selecting "not applicable" are excluded. Regional 
Center Ns range from 108-126 in 2015; 100-108 in 2016; and 95-110 in 2017. Content Center Ns range from 68-75 
in 2015; 87-95 in 2016; and 106-113 in 2017. 
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Exhibit D-15. TA Recipients’ Item-Level Ratings of Centers’ Contributions: 
Organizational Capacity 


To what extent do you agree that the technical assistance you Percentage of Respondents Reporting 
received during the last 12 months helped your organization in the “To a Moderate Extent” 


following ways? OR “To a Great Extent” 
All Regional | Content 
Centers | Centers | Centers 


Partner more with external experts 177 86% 83% 
Identify other needs for information 130 83% 79% 
Incorporate the new information into internal procedures and processes | 172 80% 69% 
Improve communication channels across state agencies 143 69% 67% 
Develop and implement internal staff trainings 159 71% 57% 
Redesign state agency structures, teams, and departments 140 54% 54% 
Centers | Centers | Centers 


Partner more with external experts 179 91% 
Identify other needs for information 136 85% 
Incorporate the new information into internal procedures and processes | 168 83% 
Improve communication channels across state agencies 162 75% 
Develop and implement internal staff trainings 164 76% 
Redesign state agency structures, teams, and departments 147 61% 


Centers | Centers | Centers 
Partner more with external experts 201 87% 
Identify other needs for information 148 87% 
Incorporate the new information into internal procedures and processes | 196 81% 
Improve communication channels across state agencies 178 76% 
Develop and implement internal staff trainings 178 71% 
Redesign state agency structures, teams, and departments 163 56% 


Source: TA recipient survey. 

Total N = Number of respondents selecting a rating; respondents selecting "not applicable" are excluded. Regional 
Center Ns range from 77-111 in 2015; 74-100 in 2016; 76-100 in 2017. Content Center Ns range from 53-66 in 2015; 
62-79 in 2016; 72-103 in 2017. 
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Exhibit D-16. TA Recipients’ Item-Level Ratings of Centers’ Contributions: 
Capacity for Policy Design 


To what extent do you agree that the technical assistance you Percentage of Respondents Reporting 
received during the last 12 months helped your organization in the “To a Moderate Extent” 
following ways? OR “To a Great Extent” 


Total Regional | Content 
2015 
N Centers | Centers | Centers 
Shape new and existing programs and policies in the area of great 35% 90% 17% 
teachers and leaders 
Shape new and existing programs and policies in the area of early 
iearaing 65% 69% 61% 


Content 
Centers 


Shape new and existing programs and policies in the area of great 


83% 
teachers and leaders i 


Shape new and existing programs and policies in the area of early 65% 
‘0 


All Regional | Content 
Centers | Centers | Centers 


Shape new and existing programs and policies in the area of great 

P ae : 88% 90% 86% 
teachers and leaders 
Shape new and existing programs and policies in the area of early 69% 7A% 64% 


learning 


learning 


Source: TA recipient survey. 

Total N = Number of respondents selecting a rating; respondents selecting "not applicable" are excluded. Regional 
Center Ns were 58 (shape new policies in early learning) and 107 in 2015; 66 and 96 in 2016; 78 and 99 in 2017. 
Content Center Ns were 56 for both items in 2015; 68 and 77 in 2016; 75 and 95 in 2017. 
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Exhibit D-17. TA Recipients’ Item-Level Ratings of Centers’ Contributions: 
Capacity for Policy Implementation 
To what extent do you agree that the technical assistance you Percentage of Respondents Reporting 
received during the last 12 months helped your organization in the “To a Moderate Extent” 
following ways? OR “To a Great Extent” 


Regional | Content 


Improve communication channels with LEAs 


Provide better support to LEAs in the area of great teachers and leaders 
Provide better support to LEAs in the area of early learning 


Regional | Content 
Centers | Centers | Centers 
Regional | Content 
Centers | Centers | Centers 


Improve communication channels with LEAs 
Provide better support to LEAs in the area of great teachers and leaders 
Provide better support to LEAs in the area of early learning 


Improve communication channels with LEAs 


Provide better support to LEAs in the area of great teachers and leaders 


Provide better support to LEAs in the area of early learning 

Source: TA recipient survey. 

Total N = Number of respondents selecting a rating; respondents selecting "not applicable" are excluded. Regional 
Center Ns range from 53-106 in 2015; 60-99 in 2016; and 68-91 in 2017. Content Center Ns range from 48-52 in 
2015; 55-64 in 2016; and 67-85 in 2017. Lowest Ns for both types of Centers were for “Provide better support to 
LEAs in the area of early learning.” 


NCEE 2020-001 National Evaluation of the Comprehensive Centers Program D-17 


TA recipient survey respondents were asked to select from a list of actions in which they might 
have engaged using the Centers’ TA. Exhibit D-18 shows that results were similar across years, 
although overall reported use of Centers’ information in developing new processes/ 
projects/programs, in developing drafting internal memos/plans/reports, and to overcome a 
barrier or challenge declined somewhat after 2015. 


Exhibit D-18. TA Recipient Actions Reported, by Year 


TA Recipient Actions Using Materials Provided by the Centers Percentage of TA Recipients 
a Regional Content 
2015 Total Centers Centers 
N=133 N= 82 
Made recommendations to SEA leadership or staff 61% 56% 70% 
Used the information in developing new processes/projects/programs 68% 15% 57% 


Used the information in drafting internal memos/plans/reports 63% 68% 55% 


Used the information to overcome a barrier or challenge 52% 56% 45% 


Reported at least one of the above actions 37%| 90% 82% 


a Regional Content 
2016 Total Centers Centers 
N=112 N= 102 
Made recommendations to SEA leadership or staff 63% 63% 63% 
Used the information in developing new processes/projects/programs 58% 76% 39% 


Used the information in drafting internal memos/plans/reports 61% 71% 51% 


Used the information to overcome a barrier or challenge 43% 51% 33% 


Reported at least one of the above actions 94% 81% 


Regional Content 
ylihly{ We) ¢-] | Centers Centers 


N=231 N=115 N=116 


Made recommendations to SEA leadership or staff 64% 61% 66% 


Used the information to develop new processes/projects/programs 63% 69% 58% 
Used the information in drafting internal memos/plans/reports 57% 64% 50% 


Used the information to overcome a barrier or challenge 46% 50% 42% 


Reported at least one of the above actions 86% 89% 84% 
Source: TA recipient survey. 
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Outcome Measures Used by Centers 

Evaluators used interview data and review of Centers’ Year 4 evaluation reports to examine 
outcome measures used by the Centers.”4 Centers’ evidence for having achieved the project 
outcomes they described or reported included a combination of formal methods, such as 
surveys and interviews, and informal methods, such as ongoing discussions, project progress 
tracking, and reflections. 


Center interview respondents were most likely to identify the use of surveys (86% of projects) 
and informal methods (89%) as the sources of evidence for having achieved project outcomes 
(Exhibit D-19). Informal methods included informal observations of constituent activities, 
progress tracking, conversations with constituents, and team reflections. Over two-thirds of 
Center respondents (67%) identified interviews as an approach used in measuring outcomes. A 
smaller percentage of respondents mentioned using case studies to aid in the measurement of 
outcomes of these projects (17%). Regional and Content Center staff reported similar methods. 


Exhibit D-19. Centers’ Sources of Evidence for Project Outcomes 
As Reported in Center Staff Interviews 


Total Content Centers 
Methods of Measuring Outcomes (N = 36 projects) (N = 25 projects) (N = 11 projects) 


Source: Center staff interviews, 2017. 


Center staff respondents described several types of informal methods for assessing project 
outcomes: informal observations, in which Center staff observed and took notes on the TA 
recipients’ progress to look for follow-through or ongoing participation (16 projects); program 
performance tracking in a Center database, which might track projects through program 
indicators or through a narrative (nine projects); informal constituent feedback through 
conversations (nine projects); and Center staff team discussions and reflection (five projects). 


Formal methods Centers used to collect data to evaluate outcomes of projects described in 
interviews included surveys, interviews, and case studies. Evaluation reports for 2016, as shown 
in Exhibit D-20, confirmed that all Centers used surveys, interviews, or both, to measure 
outcomes of their projects. Some evaluation reports also included case studies that described 
or analyzed project outcomes, especially capacity-building outcomes. 


Although surveys sometimes focused on customer satisfaction with the quality, relevance, and 
usefulness of Centers’ work (as required for Government Performance and Results Act 


4 Centers were required to submit annual evaluation reports to the U.S. Department of Education. Evaluation 
reports reviewed for this discussion were submitted in early 2017, reporting on Centers’ work in 2016, Year 4 of 
the grant. 
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reporting), they also included questions about project outcomes, including capacity-building 
outcomes. Some Centers included questions about capacity building and other outcomes that 
were designed separately for each survey or interview. Others administered capacity-building 
rubrics as part of interviews or surveys; capacity-building rubrics are assessment tools 
specifying capacity-building dimensions or goals along with indicators or performance measures 
associated with each dimension. The rubrics might be administered by evaluators, Center staff, 
or the TA recipients themselves to produce ratings on each dimension, which can be compared 
over time to measure growth. 


Exhibit D-20. Outcome Measures Used in Centers’ Evaluation Reports, 2016 


Regional Content 


Centers Centers 
(N = 15) (N=7) 


Data collection tools that included outcomes measures 


Interviews alone 3 0 


Surveys alone 


Both interviews and surveys 


Type of surveys 


Post-event 


Annual or other regular interval 8 6 


Other (post-project or pre- and post-project) 


Case studies 


Teacher/leader case studies 11 


Early learning case studies 5 0 


A capacity-building tool or rubric was included in evaluation report 
Yes 7 2 
No 4 
Data collection tools were informed by questions related 
to capacity building, but no rubric was included in the report.* 

Source: Center evaluation reports, 2016. 

*Centers may have used rubrics that were not included in the 2016 evaluation reports. 
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Effective Strategies 

In interviews, Center staff and TA recipients were asked to describe project outcomes and 
identify TA Center strategies and activities that they found to be particularly effective in 
producing these outcomes. Exhibit D-21 shows that both Center staff and TA recipients 
identified centers’ products and resources most often. Center staff were more likely than TA 
recipients to identify thought partnering, modeling, and training as effective strategies. Both 
respondent groups identified stakeholder engagement/facilitation as an effective strategy for 
over 40% of projects. 


Exhibit D-21. Effective Strategies and Activities as Reported by Interview Respondents 


Center Interviews 


otal Regional Centers Content Centers PA Bcepien 


(N = 36 - : Interviews 
: N=2 t N=11 t 
Effective Strategies projects) ( el Rulsos)] ( projects) (N = 31 projects) 
Products and resources 67% 56% 91% 74% 
Thought partnering 56% 68% 27% 35% 
In-person training sessions 44% 44% 45% 29% 
ee PHEseement/ 44% 52% 27% 55% 
Modeling 42% 48% 27% 6% 
ae mnewinoue 25% 20% 36% 35% 
Data analysis 17% 12% 27% 6% 
Linkages across policy areas 17% 16% 18% 0% 


Source: Center staff and TA recipient outcomes interviews, 2017. 
Percentages indicate proportions of projects for which the strategy was identified as effective in producing capacity 
outcomes. Respondents identified multiple strategies for some projects. 


On the TA recipient survey, respondents were asked to write responses to the open-ended 
question, “Please describe the aspects of the services and/or resources that have been most 
helpful and why?” Exhibit D-22 shows that the patterns of responses were similar across years. 
There appeared to be some differences between Content Centers and Regional Centers. For 
example, TA recipients of Content Centers were more likely than those of Regional Centers to 
identify Centers’ products and resources. 
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Exhibit D-22. Most Helpful Center Services Reported by TA Recipients, By Year 


Service or Resource 


2015 


Provision of products and resources 
Access to experts 

Responsiveness 

Meeting facilitation 

Support for product or plan 
Relationships 

Cross-state peer sharing 

Thought partnering 

Training 

No response 


2016 


Provision of products and resources 
Access to experts 

Responsiveness 

Meeting facilitation 

Support for product or plan 
Relationships 

Cross-state peer sharing 

Thought partnering 

Training 

No response 


2017 
Provision of products and resources 
Access to experts 
Responsiveness 
Meeting facilitation 
Support for product or plan 
Relationships 
Cross-state peer sharing 
Thought partnering 
Training 
No response 


Source: TA Recipient Survey. 


Percentage of TA Recipients Reporting Service or 
Resource as Among the Most Helpful 


All Centers 


(N = 215) 
26% 
20% 
15% 
13% 


9% 
9% 
8% 


7% 
6% 
31% 
(N = 214) 
34% 
20% 
10% 
11% 
12% 
11% 
8% 
8% 
8% 
27% 


(N = 231) 
33% 
20% 
11% 
8% 
13% 
9% 
10% 
8% 
14% 
29% 


Write-in responses were coded using codes described in Appendix C. 


Regional Centers 


(N = 133) 
23% 
23% 

13% 
17% 
10% 
10% 
8% 
6% 
7% 

31% 

(N = 112) 
25% 
22% 

9% 
17% 
15% 
11% 
8% 
7% 
5% 
25% 


(N = 115) 
26% 
22% 
10% 
10% 
17% 
8% 

11% 
10% 
10% 

32% 


Content Centers 


(N = 82) 
32% 
16% 
20% 
5% 
7% 
7% 
10% 
7% 
5% 
31% 
(N = 102) 
43% 
18% 
12% 
5% 
8% 
12% 
9% 
8% 
12% 
28% 


(N = 116) 
41% 
18% 
11% 
7% 
10% 
10% 
10% 
6% 
17% 
25% 


The TA recipient survey also posed open-ended questions that asked respondents to write 
about the Center services that were least helpful and about needs that were not met by the 
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Centers. As shown in Exhibits D-23 and D-24, most survey respondents either left these blank or 
responded that all services were helpful and all needs were met. About 10 to 20 percent of 
survey respondents each year affirmatively noted a problem with some aspect of Centers’ 
services. 


Exhibit D-23. TA Recipient Reports on Least Helpful Services of Centers 


2015 2016 2017 
Responses regarding least helpful services (N = 215) (N = 214) (N = 231) 


Relevance or quality concerns with certain services or 
ee 13% 12% 10% 
products 


SEA limitations made it difficult for respondent to use the 

; P 2% 3% 1% 
Center's TA 
Respondent is “Not Sure” or answer is unclear and cannot 1% 2% 3% 
be coded 


Source: TA recipient survey. 
Write-in responses were coded using methods described in Appendix C. 


Had concerns about availability, accessibility or timeliness of 5% 1% 2% 
Center staff 


Exhibit D-24. TA Recipient Reports on Needs Not Met by Center 


2015 2016 2017 
Responses regarding needs not met (N = 215) (N = 214) (N = 231) 


Would like more resources, connections, or TA in some 
a ’ , 4% 5% 5% 
priority areas 


Center staff changes caused confusion/would have liked 
more clarity or consistency 


Source: TA recipient survey. 
Write-in responses were coded using methods described in Appendix C. 


Center staff were asked on the survey to select the most significant supports (up to three) from 
a list provided for each project on which they worked (Exhibit D-25). Regional Centers and 
Content Centers had some differences. For example, Regional Centers were more likely than 
Content Centers to identify relationships with SEA staff and leadership, while Content Centers 
were more likely than Regional Centers to identify collaborations with Regional Comprehensive 
Centers. However, about half of respondents from both types of Centers identified 
collaboration with the other type of Center as a significant support (46% of respondents from 
Regional Centers identified collaborations with Content Centers, and 55% of respondents from 
Content Centers identified collaborations with Regional Centers). 
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Exhibit D-25. Project Supports Reported by Center Staff 


Percentage of Projects for Which at Least 
One Center Staff Reported This Support 
Regional Content 
Centers Centers 

(N =171) (N = 108) 

Content expertise among Center staff 87% 87% 87% 


Supports for Projects 
All Centers 
(N = 279) 


Knowledge of state context among Center staff 15% 79% 69% 
Strong relationships between Center staff and SEA staff 62% 69% 52% 
Strong relationships between Center staff and SEA leadership 58% 70% 40% 
SEA Leadership commitment to the project 49% 55% 40% 
Supportive leadership in Center 48% 46% 52% 
Collaboration with Content Comprehensive Center 39% 46% 28% 
Expertise among SEA staff 37% 41% 31% 
Collaboration with Regional Comprehensive Center 33% 19% 55% 
Collaboration with a subcontractor 14% 15% 12% 
Collaboration with a professional association (e.g., ASCD, CCSSO) | 10% 7% 15% 
Collaboration with a Regional Education Laboratory 9% 10% 8% 


Collaboration with an institution of higher education 7% 8% 5% 
Collaboration with a government agency 6% 4% 10% 
Source: Center staff survey, 2015-2017. 
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APPENDIX E: Project Profiles 


Evaluators developed two case studies to illustrate how a project’s design, implementation, and 
outcomes played out within a specific state policy context. The two projects, one in each 
priority area, were purposefully selected from the pool of projects discussed in interviews with 
Center staff and TA recipients. These projects were selected because they spanned multiple 
years and were illustrative of the various needs, strategies, challenges, and results that were 
reported for Centers as a group in the report. However, these projects are not necessarily 
representative of all projects examined in the report, and these profiles should primarily be 
viewed as a source of richer description on design, implementation, and outcomes, to 
supplement the more general findings presented in the report. 
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Mississippi Early Childhood: Southeast Comprehensive Center 


Project Overview 


Constituents Mississippi Department of Education (Office of Early Childhood, Office of Early Intervention 
Services, Office of Special Education, Federal Programs Office), Mississippi Department of 
Human Services Office of Early Childhood Care and Development 


Major partner Center for Enhancing Early Learning Outcomes (CEELO, which is a Content Center) 
State policy goals | Improve early learning statewide; ensure the success of the state-funded early learning 
collaboratives in Mississippi 


Center TA Support state-funded pre-K in Mississippi with early learning guidelines and standards, a 
objectives monitoring tool, and a continuous improvement approach to implementation of a state- 
funded early learning initiative 


Needs Assessment and Project Development 

The Southeast Comprehensive Center (SECC) began conducting needs sensing with the 
Mississippi Department of Education (MDE) at the beginning of the grant period in late 2012 to 
support the improvement of early learning programs in the state. Center staff conducted 
needs-sensing interviews with the state-level early childhood staff in Mississippi. They also used 
the results of their own needs-sensing survey and a needs-sensing survey administered by the 
Center for Enhancing Early Learning Outcomes (CEELO) to gain an understanding of the state’s 
current status in providing early childhood education and to clarify the state education agency’s 
(SEA) goals in this area. 


MDE’s goals and requests for assistance from the Center were informed in part by the state’s 
participation in an early childhood regional “community of practice,” coordinated by SECC and 
supported by CEELO. Through the community of practice meetings and online discussions with 
other states, staff of MDE and the state’s Department of Human Services gained ideas for how 
they might develop a publically funded high-quality early childhood infrastructure in 
Mississippi, and also developed a relationship with the SECC and CEELO staff. 


As the state’s plans evolved, its needs for specialized TA grew. Initially, MDE staff requested 
help from the SECC in facilitating an early childhood stakeholder workgroup and supporting the 
workgroup in reviewing and updating the state’s early learning guidelines, which at that time 
included guidelines (basic requirements regarding class sizes and teacher qualifications), 
standards (skill and learning objectives), and teaching strategies all together in one complex 
document. 


In 2013, the state passed legislation to fund community-based collaboratives expected to start, 
support, and expand early childhood education services.2? When MDE created its first Office of 
Early Childhood in 2014, and officially hired a director in 2015, the new staff identified 

emerging needs for building the state’s early childhood education infrastructure and improving 


25 Mississippi passed the Early Learning Collaborative Act in 2013 and funded the first round of 11 collaboratives to 
support and facilitate the implementation of pre-K programs. Today, there are 14 collaboratives across the state. 
See http://www. mississippifirst.org/education-policy/pre-kindergarten/early-learning-mississippi/. 
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the implementation of the collaboratives. While SECC continued to assist with the revisions to 
the early learning guidelines and standards, the Center also offered support for selecting an 
early learning assessment, creating a tool to help MDE monitor implementation of the 
collaboratives, and developing a continuous quality improvement process for the 
collaboratives. Later, the Office of Early Childhood director also requested help with her work 
related to developing a high-quality early childhood workforce. 


The Office of Early Childhood director explained that the SECC staff communicated regularly to 
ask about her needs and she “had a list for them.” Because MDE had never before had an Office 
of Early Childhood, “we just had this long list of things that we needed,” including information 
about “research, policy, definitions, and other states’ plans.” When working with the 
stakeholder group on the development of plans or proposals, the director would email the SECC 
and CEELO as needed and “in a day or so, we were able to get a document that would help 
support the work.” 


Center TA Strategies and Activities 

Initially, at the request of the Chief Academic Officer at MDE, the SECC took the lead on 
facilitating the meetings of MDE’s early childhood stakeholder workgroup, an inter- 
organizational group focused on the goal of providing high quality early childhood education 
programs across the state. The Center provided support for MDE’s and the workgroup’s 
standards development and also developed additional resources for them to use. In SECC’s first 
year, they reviewed alignment of the MDE age 3 and 4 early learning standards with the Head 
Start early learning framework and recommendations from the National Association for the 
Education of Young Children and also with MDE’s K-12 standards. The Center also provided 
“mastery examples” —examples of child skill mastery—to guide teaching in six of eight teaching 
domains aligned with the early learning standards. In the second year, SECC continued to 
facilitate stakeholder meetings, focusing on the implementation of the early learning standards 
and teaching domains, and produced mastery examples for the final two teaching domains. 


In SECC’s third year, MDE hired a director of the Office of Early Childhood who took over the 
facilitation role for the stakeholder workgroup, so the Center’s role changed to that of being a 
thought partner for the new director. As such, the Center provided consultation on issues 
identified by the director, including continuous improvement of the collaboratives and other 
early childhood services, research on best practices, and examples from other states. To 
support continuous program improvement, SECC staff reviewed and guided the design of a 
program monitoring tool and processes for providing programs with feedback and coaching. 
SECC and CEELO also provided information on high-quality early childhood programming, how 
to define school readiness, and how to select an early learning assessment. 


In the Centers’ fourth year, SECC assisted MDE in bringing teachers and administrators into the 
early childhood stakeholder workgroup meetings, and revising the early learning standards and 
mastery examples based on their input. SECC, CEELO, and the MDE director of early childhood 
continued to meet monthly to brainstorm on implementation of these standards. 
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Over time, the Center’s work expanded to include educator workforce development. MDE and 
SECC worked collaboratively with institutions of higher education to align early childhood 
teacher preparation programs with the early learning standards. To support this collaboration, 
SECC supported CEELO with facilitating a “learning table” (learning community) that brought 
universities and MDE together in discussions. 


Project Challenges and How the Center Addressed Them 
The SECC and MDE faced challenges on this project. The challenges and the Centers’ approach 
to addressing them include: 


e Changing priorities, cuts in MDE staffing/budgets, and timeline constraints. MDE staff 
faced time, budgetary, and staff constraints. They also dealt with competing priorities 
among early learning staff, especially with Every Student Succeeds Act (ESSA) planning 
going on at the same time. The Center adjusted the pace of work to accommodate MDE 
staff availability and provided resources, such as guidelines for supporting students with 
disabilities and dual language learners, and summaries of early childhood policies in 
other states, to save MDE staff time. Center staff supported MDE staff to write a grant 
for more staffing and helped them to delegate tasks. 

e Limited Center resources and meeting diverse needs. The SECC staff member with the 
greatest early learning expertise was also working on several other projects and had 
limited time available. SECC brought in their Mississippi state liaison and subcontractor 
staff to adequately support the early learning work with MDE. The collaboration with 
CEELO also enabled SECC to draw on additional staff and resources. 


Effective TA Strategies and Activities 

Center staff and TA recipients reported that the TA strategies and activities listed below were 
particularly effective. Through these strategies, the Center served as a flexible thought partner 
and facilitator of stakeholder workgroup meetings, provided tools, and helped overcome 
project challenges. 


e Meeting facilitation and stakeholder engagement, especially by increasing the number 
of teachers on the stakeholder workgroup. TA recipients reported appreciating that 
facilitators from the SECC treated them as intelligent professionals who, with the right 
information, could solve their own problems. Workgroup members were encouraged to 
come up with solutions and measure the results. TA recipients also appreciated that 
SECC staff could be available in-person for meetings and consultations. 

e Modeling facilitation strategies. The SECC modeled facilitation strategies for the early 
childhood director and gradually turned facilitation of the stakeholder workgroup 
meetings over to the director. As one TA recipient explained, the Center would also 
model how district leaders could engage with their peers over the data. 

e Cross-Center collaboration to provide extensive early learning resources and cross- 
state knowledge building. The SECC and CEELO provided MDE and the stakeholder 
workgroup with resources on early learning standards and assessment, guidance 
materials, and continuous improvement processes. Some of these materials drew on 
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evidence-based early learning practices from other states. The Center also provided 
opportunities for SEA staff to meet with their counterparts from other states as well as 
provided state agencies’ contact information, so SEA staff could initiate conversations as 
needed. 

e Thought partnering and ongoing consultation. Center staff said that monthly calls 
between SECC staff and SEA staff were effective in building a collaborative relationship. 
TA staff reported that they valued the availability and the quality of the information 
provided, and used the information to inform their work. 


Project Outcomes 

Both the Center staff and TA recipients reported that Center work had enhanced MDE staff’s 
knowledge and skills, and organizational, policy design, and policy implementation capacity in 
the early learning area. In addition, the Center helped MDE achieve the goals of its initiative. 
Center staff and TA recipients credited the Center’s TA with the following project outcomes: 


e Knowledge and skills. Center staff reported MDE staff gained content and policy 
knowledge, greater awareness of the complexity of the work, and learned how to plan. 
TA recipients reported gaining knowledge, with the Center’s help, of early learning 
policies in other states that informed their approach. 

e Organizational capacity. Center staff reported that MDE refined its internal division of 
labor for early learning, which increased interagency communication and collaboration. 
Both Center staff and TA recipients reported that MDE developed its capacity to involve 
stakeholders in decision making; MDE recruited stakeholders and established a 
facilitated process to obtain their input. 

e Policy design capacity. Center staff said MDE implemented a process to review and 
refine documents, and noted that MDE had received national recognition for aligning its 
early learning and ESSA plans. TA recipients reported MDE used information provided by 
the Center and its partners about other states to avoid mistakes in policy development, 
and that MDE worked with Head Start and other childcare providers in the state to 
bridge early learning policy and increase alignment. 

e Policy implementation capacity. Center staff reported MDE carried out a deliberate, 
inclusive, and staged rollout of the early learning policy and implemented a continuous 
improvement process. TA recipients said that MDE is still using its Center-influenced 
continuous improvement process. 


In addition, the TA recipients reported that the Center helped them achieve state policy goals. 
With materials and support provided by the Center, MDE implemented policies, procedures, 
monitoring tools, and assessments to evaluate and continuously improve collaboratives and to 
position the program for growth. In SECC’s fifth year, the state early learning collaboratives met 
all 10 benchmarks for minimum state preschool quality standards from the National Institute 
for Early Education Research. Center-provided support and tools supported this achievement. 
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Arizona Educator Effectiveness: West Comprehensive Center 


Project Overview 


Constituents Arizona Department of Education (ADE) 

Major partners Regional Educational Laboratory (REL) West, Center for Great Teachers & Leaders (GTL, 
which is a Content Center) 

State goals Strengthen educator workforce, identify needs of teachers and leaders for support and 


professional development; ensure equitable access to effective educators; develop 
strategies for recruitment and retention to address a severe teacher shortage and 
shrinking pipeline 

Center objectives Build knowledge of evidence-based practices for improving teacher effectiveness, build 
organizational and policy capacity to help ADE better organize itself and use its resources 
effectively to accomplish state goals 


Needs Assessment and Project Development 

Staff of the Arizona Department of Education (ADE) participated in a regional collaborative on 
educator effectiveness organized by the West Comprehensive Center (WCC) to identify and 
address common needs. WCC brought state education agency (SEA) teams from five states 
together two to three times per year (beginning in 2010), with additional meetings or webinars 
as needed to focus on state priorities such as educator evaluation systems and the 
development of equity plans or Every Student Succeeds Act (ESSA) plans. WCC supplemented 
the collaborative meetings with calls or meetings with SEA staff and chief state school officers 
to identify state-specific needs, and to plan and implement Center TA for each state. 


Center staff met each year with the Arizona superintendent, associate superintendents, and 
SEA lead staff to review the state’s priorities, identify how WCC would assist ADE, and develop 
specific action plans to guide the Center’s work on teacher effectiveness with the state. Each 
annual plan was treated as a “living document,” according to one SEA staffer, and was adapted 
throughout the year as needed. Over time, WCC also helped ADE staff identify needs based on 
data such as results of pilot studies of the new teacher evaluation system, and an examination 
of geographic distribution and trends in teacher shortages throughout the state. 


ADE initially requested help with improving its educator evaluation system to meet state 
statute, state board policy, and Race to the Top requirements. Over time ADE also asked WCC 
for advice or assistance in implementing a new evaluation system with local education agencies 
(LEAs), especially by training evaluators, and using evaluation data for feedback and teacher 
professional development. ADE staff also asked for TA to help them plan equitable access to 
high-quality educators across schools and to consider new initiatives on recruitment and 
retention to address the state’s teacher shortage. 


Center TA Strategies and Activities 

Between 2014 and 2016, WCC facilitated a pilot study of the implementation of Arizona’s state 
teacher evaluation framework. The Center created an initial plan for the study and brought in 
REL West researchers to develop the study design and do the analysis. WCC collected the 
interview and survey data used in the study from districts and schools, worked with REL West 
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to report on findings, and worked with ADE to develop lessons and recommendations based on 
the study. Study findings were used to inform the training of principals and other evaluators. 
Each year, the Center also worked with ADE to plan an educator summit on teacher/leader 
effectiveness, bringing together about 400-500 LEA staff from around the state. The Center 
brought in speakers and facilitated sessions at the summit. 


Beginning in 2016, the Center supported ADE in delivering a “Qualified Evaluator Academy” for 
principals, assistant principals, and other educators involved in educator evaluation. WCC staff 
sat in on sessions and gave feedback to ADE. The Center planned and led a workshop for ADE 
and LEA staff on using educator data to design “talent management,” including decision-making 
about coaching and professional development, retention, and promotions. The Center 
facilitated a retreat with staff of ADE’s Effective Teacher and Leader Division to discuss division 
goals and objectives, strategies, expected outcomes, and performance metrics. 


Also in 2016 WCC invited ADE staff to join a cross-regional state collaborative led by the GTL 
Center, which brought together states that had developed evaluation systems and sought to 
share information about lessons learned and generate ideas on how to move forward with the 
next iteration of these systems. WCC continued to follow up with ADE staff on these issues 
between meetings. 


WCC worked with all of the states in the region on their equity plans during regional meetings, 
and consulted independently with ADE by helping them review data on teacher shortages and 
connecting them with REL West to examine state workforce trends. The Center also helped ADE 
plan meetings with LEAs to discuss implementation of equitable access including how to retain 
quality educators through opportunities for career advancement and leadership development. 


Upon the passage of ESSA, WCC met with the ADE superintendent, associates, and staff to work 
on development of the state ESSA plan. The Center led several discussion sessions with ADE 
staff to help them determine how they would define teacher effectiveness, address equitable 
access, and guide LEAs in implementing evidence-based strategies. 


Project Challenges and How the Center Addressed Them 
The WCC faced challenges on this project and addressed these challenges as detailed below: 


e SEA turnover and cutbacks. During the course of the project ADE experienced staffing 
and leadership changes, including a new superintendent. This turnover challenged the 
momentum of the project. ADE also experienced staff cutbacks. One ADE staff member 
said, “I had to cut four positions last year and then [for this year] four more positions ... 
So | went from five directors to two directors and from ten program specialists to six 
program specialists, yet the work has increased due to the dire needs of effective 
educators.” WCC and TA recipients described the Center’s consistent communication 
and follow-through as critical to sustaining effort throughout SEA turnover and 
cutbacks. A TA recipient said, “They are the solid steel cable that keeps us all connected, 
moving on the same track, getting us up to speed so that there are no glitches.” Another 
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TA recipient said, “In a very proactive collaborative way, they contact me frequently so | 
don’t have to contact them.” A TA recipient commented that the Center helps by “giving 
us history, helping us look at the initial goals, keeping us on that same path, with some 
tweaking.” Center staff said, “We understand the SEA job and how difficult it is. All 
Center staff previously worked for an SEA.” 

Shifting priorities. ADE’s priorities shifted over time due to changing educator 
effectiveness requirements under ESSA and challenges posed by the state’s teacher 
shortage. As the teacher shortage became more severe in Arizona (with projected 
greater shortages in coming years), priorities shifted from focusing on improving 
educator effectiveness to trying to maintain and improve it at the same time; as one TA 
recipient put it, “trying to keep current staff [of schools] as well as trying to improve the 
effectiveness of staff.” Both Center staff and TA recipients reported that the Center 
responded promptly, even proactively, to shifting priorities. The Center staff also noted 
that the WCC supported ADE in using data to improve the quality of the existing 
workforce and to develop a leadership pipeline “moving teachers up, not out.” 


Effective TA Strategies and Activities 

WCC staff and TA recipients reported that the Center’s consistent communication, long-term 
relationship with ADE, access to expertise, and skill in bringing stakeholders together were 
particularly effective in building ADE’s capacity and advancing their work. Among strategies that 
respondents highlighted were: 


Thought partnering through consultations and data review or analysis. Center staff 
maintained ongoing close communication with key ADE staff, through monthly calls at a 
minimum, increasing to weekly calls during periods of high activity. The Center analyzed 
state documents, task force reports, research reports, and evaluations to help the SEA 
set priorities and use data. 

Cross-state knowledge building and access to expertise. TA recipients valued the 
connections to other states and to a variety of experts provided by WCC. Cross-state 
sharing enabled ADE to come up with ways to address gaps in their evaluation system. A 
TA recipient said, “The Qualified Evaluator Academy would never have happened except 
that this idea came up in one of these meetings, talking with other states,” because the 
states shared common problems and brainstormed about how to provide leadership 
training at scale. ADE staff also noted the quality of expertise available to them through 
the Center, which provided experts from WCC, GTL Center, and REL West, as well as 
from other states, to do presentations and training (and in the case of REL West, to 
conduct pilot studies). 

Cross-policy coordination through meeting facilitation. The Center emphasized 
breaking down silos and fostering cross-team and cross-agency communication within 
the SEA and with other partners. WCC brought together school improvement, Title | 
special programs, and teacher/leader effectiveness teams. As one TA recipient said 
“They [Center staff] have a way of getting us all together in the same room to talk about 
common needs and strategies ... because they know us and what our challenges are, 
and we know them.” 
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Modeling. The Center described “co-construction” of the educator summit, ultimately 
leading to ADE being able to take over leadership of the summit. The Center first took a 
strong lead, and gradually the SEA staff learned the process, the structure, and ways to 
facilitate and organize the meetings. 


Project Outcomes 
Center staff and TA recipients credited WCC’s TA with contributing to multiple areas of SEA 
capacity, including: 


Knowledge and skills. SEA staff learned to “identify problems in their system that they 
need to address, and come up with actions to address those,” as reported by Center 
staff. TA recipients said the Center activities built knowledge among a core group of ADE 
staff, and that their knowledge advanced because the Center provides “another set of 
eyes and ... we get new ideas from talking to other states.” 

Organizational capacity. Both Center staff and TA recipients stated that the Center 
contributed to improved coordination across SEA programs and divisions. ADE began to 
integrate educator evaluation across other areas; for example, school improvement 
policy now includes observation and feedback, educator support, and data use to inform 
professional learning. Based on WCC’s work, ADE began realigning systems of support 
for low-performing schools and educator effectiveness, creating a coordinated system 
at the state level so that LEAs have one place to go for questions about initiatives in 
these policy areas. Because multiple people were involved in making changes, capacity 
was built across the agency, beyond a few individuals. 

Policy design capacity. Center staff said the SEA improved its capacity to identify and 
address gaps in systems. ADE staff realized that a “gaping hole was support for 
principals; they began planning for building the capacity of principals to conduct 
effective observation and feedback to promote professional learning and growth.” 
Center staff also reported that ADE staff learned to continually re-assess their systems 
and improve them. As ADE staff realized they couldn’t directly train all principals in 
evaluation, they developed a strategy for working with districts so districts can train 
their own principals. 

Policy implementation capacity. Both Center staff and TA recipients agreed that ADE 
improved its capacity to provide technical support to LEAs on teacher evaluation so that 
“we can have this implementation that is valid and reliable,” as one staff commented. 
The SEA also increased its outreach to principals in the state, “supporting districts in 
order to support principals.” A TA recipient said, “The [Center] significantly impacted my 
division’s ability to support educators” in implementing an evaluation system, 
“thoroughly understanding it, and ... embedding it in the culture.” 


SEA staff also reported that the Center’s TA was integral to helping the state meet its 
teacher/leader effectiveness goals. In particular, WCC’s TA helped the state make progress in 
improving their educator evaluation system and its use for professional development. The state 
met Race to the Top requirements through adoption of a rigorous evaluation model. Beyond 
meeting the legal requirements, TA recipients emphasized that the Center helped them 
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develop an understanding among educators across the state that “this isn’t just a compliance 
piece, it is about improving teacher practice.” Training and support for evaluators has included 
at least 150 participants and is continuing, working on honing principals’ skills for gathering 
data, analyzing it, and having conversations about it with teachers focusing on professional 
learning and growth. SEA staff also reported that WCC’s facilitation improved their ESSA 
planning and helped ensure that ADE can meet their ESSA goals. 
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